As a matter of fact, it was viable to run a factory with slaves; slavery is a stunningly versatile labor system, with various ways for people to monetize their labor and different products for them to make.

The rate of returns on industrial slave labor in the antebellum south was solidly in the profitable range (you need something like six percent returns to be viable, and the average of the industries they looked at was about eight percent), and the work they were doing ranged from mining, to ironworking, to railroad construction, working in distilleries, sawmills, flour mills, and similar industries. You get the most out of the system when slaves are doing completely unskilled work, but slaves worked a wide variety of skilled and unskilled trades throughout the south, as blacksmiths, sailors, stable keepers, and so on. Furthermore, the common practice of renting out slave labor compounded this versatility.

Virginia is a good example of slavery's versatility. In 1860, it was the largest wheat producing state in the Union. Thing is, you only really need lots of labor for wheat around harvest time, so for the rest of the year, slaveowners had lots of slaves they could lease out to renters ranging from farmers too poor to buy their own slaves to railroad companies needing to expand their labor pool for a job. This practice lets slavery mimic the economics of free labor in many ways; they're not just stuck in whatever jobs are available on the plantation, but can be assigned to whichever job yields the greatest returns.

That aside, we have to turn the clock back to the late 18th century. Here, the viability of slavery as an institution is tied to the availability of a cash crop that can be sold on international markets. In the 18th century, that crop was tobacco, and to a lesser extent rice and indigo. Gradually, however, tobacco became less profitable, as more people got in on the game, and the soil in the Upper South was exhausted by intensive tobacco cultivation. In the north, it never really took off, because the climate wasn't generally suited to producing cash crops. They had slavery, but it was very much marginal to the economy. Many thought slavery in the late 18th century was a dying institution; states north of the Mason Dixon line ended slavery when it was at a low ebb of profitability, and lacking a reason to keep it, many thought the South would soon follow suit. The Constitution included a clause giving congress the power to end the importation of slaves in 1807.

However, there would be a new birth of slavery in the early 19th century, when the cotton gin and new varieties of cotton came onto the scene. With labor costs drastically reduced by the new device, which quickly combed the seeds from the cotton fibre, cotton became stupendously profitable; until the 1850s, the South had a near total monopoly on cotton, probably the most valuable commodity in the 19th century world. This was a global phenomenon. Cotton would be grown in the U.S. South using slave labor, ginned into bales, and shipped to either the North or Great Britain, where it would be spun and woven into textiles. These could be sold in the North, in the South, where they might clothe the very slaves that grew the cotton, or even places as far as China, where the balance of trade in textiles an green tea was a matter of grave concern for the British.

About 80% of the U.S.'s exports in 1860 were the products of slave labor. Northern factories bought their raw materials in the South, and the profits of this enterprise would go into banks, and insurance companies, who insured the slaves, who were after all considered a form or property. By 1860, the only thing more valuable than the auction price of the country's slaves was the total value of all the land in the country. That only really tells part of the story, though, since much of that land was only as valuable as it was because you could use slaves to grow cash crops on it, and the cash value of those crops would more than pay for any individual slave.

I think in a very real sense, you could say the North did need slavery; it needed a cheap way of producing the raw materials it would process into marketable goods, but the means to do so, slavery, had by historical circumstances been relegated to the U.S. South, where it had become 'the greatest material interest in the world.'

I'd recommend

The Economics of Industrial Slavery in the Old South

In the Presence of Mine Enemies by Edward Ayers, with accompanying data on his Valley of the Shadow site (look up 'The Difference Slavery Made')

Empire of Cotton by Sven Beckert is an incredible global history of the commodity

Also check out Stephen Deyle's work on the domestic slave trade