Growing up in Massachusetts, this is something I've always wondered. Signs of historical agriculture are everywhere, from old rock walls that were farmer's land boundaries, to old cisterns and wells. The soil is famously rocky in the region so we were taught that agriculture largely shifted to more prosperous areas, which made sense, but did people simply abandon their land? Sell it to new owners who found little to do with it in the emerging regional economy? Was there depopulation out of the region or within certain areas?
So I actually wrote an answer on this topic a while back. I also was taught "the soil was rocky so all the farmers moved to the Midwest", and was surprised to learn that this all actually comes from a 1927 article that has largely been debunked.
The decline was much more gradual: people (especially young people) moved to industrializing cities starting in the mid-19th century, but agricultural production and land use actually peaked late in the 19th century, and even by 1945 agriculture was still a major use of land. The crash in agricultural land use is mostly a post-World War II phenomenon, and linked to the urbanization (and suburbanization) of the period.