It's hard to answer this definitively. But there were obvious factors. First, factories required a concentrated mobile workforce ( like the unemployed farm girls moving to Lowell, MA), power sources like watermills, and decent infrastructure like roads, canals, etc. to get supplies in and products out. The sparse settlement in the rural South, and the lack of good roads, meant all that was more difficult.

Second, at the time the North was growing its industry in the 1840-1850's, the South itself had a huge growing demand for its cotton, and with the new economies of scale from its large captive workforce, large tracts of land and the recently invented cotton gin , it could meet that demand. It also had something of monopoly on the crop, and so could get a good price. A plantation could therefore bring in pretty good profits. There was also the rise in the value of the slaves themselves, which could be used as capital for securing loans, or sold. There is a factual basis for that literary character, the riverboat gambler : plantation owners or their sons would sometimes return home, after selling their crops in New Orleans, with thousands of dollars in their pockets and they could be easy marks. A Southern businessman might say, reasonably, why bother to build a factory? Business is good.

There were Southern cities, and some with factories, and some slave labor did happen. The Tredegar Iron Works in Richmond used slave labor (as well as employing free blacks ) after 1847. Some slaves were trained in trades- a slave trained to fix shoes or be a carpenter could be hired out and make more money for his master. But slaves were more typically unskilled. Harper's Ferry had a significant number of slaves ( John Brown was hoping they would revolt) but the most they did in the Armory was apparently clean and sweep floors.