There were various factors. Circa 1830, New England had greater population density, better roads, and good waterpower sites. After 1830 New England farmers also found themselves competing more and more with huge farms in the northwest ( like Ohio) when the Erie Canal made it possible to load grain onto boats and send it via the Great Lakes and the Hudson to the big cities on the Atlantic coast, and many of the farm girls who came to work in the Lowell Mass. textile mills had good reasons for leaving home.

And certainly agriculture was very profitable in the South. Tobacco ( in the 17th-18th centuries) and later cotton grow best in the warmer South, were cultivated by cheap slave labor, and there was also a great export demand for both. That made some very wealthy plantation owners.