This is an extremely hard to pin down question because New England did not have a uniform economic experience. If you can give me the town or at least the rough area I can probably talk more about it more specifically. Unfortunately, a general economic history of New England is going to be dominated by discussions of the major cities.

So I was going to give a whole answer along the lines of "New England farming communities declined in the mid-19th century because people abandoned the rocky soil and moved to the Midwest", but I decided to do a little research before writing that, and interestingly enough that's apparently a myth!

That idea that I was going to cite comes originally from a 1927 article by James Walter Goldthwait in the Geographic Review: "A Town That Has Gone Downhill". This was apparently a big theory that Goldthwait, John D. Black, and other contributors to the New England Studies Program at the American Geographical Society developed between 1927 and 1950. It emphasized that the soil in New England is rocky and thin, that most agriculture in the region was subsistence based and labor intensive (the famous stone walls being the product of back-breaking work to clear stones out of fields). The idea was that after the Erie Canal opened in 1825, and Midwestern regions were developed as agricultural producers, and a decline set in among New England farmers somewhere between 1830 and 1850.

Apparently there were real deficiencies with this theory, however. First, it tended to flatten wide varieties of agricultural land in New England: not all of it is stony hillsides (the Connecticut River Valley is practically Mid-Atlantic, to the point of producing tobacco). Nor was all of the land in agricultural use actually tilled: apparently something like a third of agricultural land was tilled, a third used for pasturage (think Vermont cows), and the rest woodland or fallow land. So much of the rocky, hilly areas were used in the 19th century for sheep and wool production.

Furthermore, apparently actual agricultural figures from the late 19th and early 20th century do not bear out the decline model: yields per acre in New England were among the highest in the country, with produce even being exported to other regions (apparently Boston-area vegetables were sold in winter markets in late 19th century Florida). Fertilizer use on New England farms was high (but not higher than in the rest of the East Coast).

Agricultural land use in New England seems to have peaked around 1880 (around 50% of the total), before showing a small decline in the late 19th and early 20th century, but even during World War II the total land under agricultural use was some 35%: after 1945 it crashed to around 10%.

Of course, as u/Erusian notes, it was heavily variable by area: even in the 1840s, there was noting of some Vermont towns going into decline, but this was very much in more marginal areas. So what changed, even if the actual agricultural industry didn't change, and if anything flourished after 1860?

Michael Bell notes that, especially in the 1850s, there was a perceived cultural shift in how farmwork was valued in rural New England. To be blunt: young people started viewing it as lame and boring, especially with rapid urbanization happening relatively close to their rural homes. Bell calls this the shift of farmer from yeoman to rube.

An 1858 pamphlet "Farm Life in New England" noted that "the farmer's life and farmer's home generally are unloved things", while Albert Comings told the Connecticut Valley Agricultural Society in 1853:

"Farming is now unpopular with the young men ... it is also with the young women. They have come to associate the name of farmer with ignorance, with stupidity, with clownishness."

It's worth noting that city life and industrialization were also much more of a permanent life choice in the period after 1860 than before. For example, the Lowell textile mills in the 1820s-1840s employed the famous Lowell mill girls, who were generally young and unmarried (presumably as they got older, they would marry and move to oversee households on farms) - industrial work was seen as a temporary career option. As industrialization and urbanization really kicked into high gear, there simply were more options for young people moving away from the family farm. It should be no surprise that Horatio Alger (a Massachusetts native) started writing his famous rags-to-riches stories in 1866.

Source

Michael Bell. "Did New England Go Downhill?" Geographical Review, Vol 79, No. 4 (Oct 1989). Available here