Antebellum textile manufacturers and bankers were certainly heavily involved in the antebellum South's economy. Cotton was, according to some historians, the single most important commodity in the antebellum American economy and the wider atlantic, even global economy as well. So certainly textile manufacturers, not just in the north but in Europe as well, profitted handsomely from the labor of enslaved men, women, and children in southern cotton fields.

Not just textile manufacturers but banks as well. The growth of the Southern economy depended heavily on extensions of credit for the purchases of more land and more enslaved people to work that land. President Jackson's enforcement of the 1830 Indian Removal Act had opened up incredible amounts of land in the then-southwest to purchase and by white settlers and enslavers. But even wealthy slaveholders tended to be rich in assets rather than cash, and there was great need for credit all around to make new purchases to exploit new potential profits in cotton growing. Cotton was fabulously profitable, and through a campaign of systematic torture of enslaved field workers (pushing them to ever-greater per-person efficiency through imposing ever-greater quotas enforced through the threat of more torture, usually in the form of whipping but in more gruesomely creative methods as well) slaveholders were ensuring an ever greater per-person yield of cotton on plantation labor camps. We might reasonably imagine that a massively expanding supply of a commodity could not possibly lead to greater profits forever (because demand cannot possibly keep up forever) but in the early 1830s, it seemed that way. Cotton producing slaveholders were in a state of "perfect competition" where they felt no need to be secretive about innovations in efficiency because the market was so demanding of their product. Bankers, recently deregulated by Jackson's destruction of the "Monster" National Bank, responded predictably, with extremely generous extensions of credit to enslavers, because they believed that profits could only grow. Indeed they developed innovative forms of credit extension to slavedriving entrepreneurs, with new systems of mortgages (on land and enslaved human beings) and risk spreading to protect their investments as much as possible. For a while, banks all over the United States (including a mass of new state banks in the South who collected a great deal of cash from the reserves of the broken-up National Bank) profited immensely from the expansion of slavery. Then the panic of 1837, caused in a major way by this over extension of credit, made a huge mess of things but you're not asking about that panic so I won't digress.

So, that is to say, you are correct that bankers and industrialists in the North profited from the products of slave labor, and from the financial systems that facilitated the enormity of the expansion of chattel slavery and the interstate slave trade (just one fact here: about 50 percent of all purchases or sales of people in the antebellum period involved breaking up a family. Some historians have called this "a kind of murder" as the massive geographical expansion of slavery in the 19th century meant that separated family members were unlikely to ever see each other again, though there are exceptions). Northern consumers benefited immensely from the wide availability of cotton. Northern women were employed making straw and palm hats to send to slaveholders that covered the heads of enslaved people harvesting cotton in the brutal summer heat. These are only a few of the ties of complicity between the North and the slaveholding South.

BUT

The idea that it was only Northerners who profited from slavery is very misguided. In 1860, the eight wealthiest states in the Union, "ranked by wealth per white person, were South Carolina, Mississippi, Louisiana, Georgia, Connecticut, Alabama, Florida, and Texas." (Source: Edward Baptist, The Half Has Never Been Told pg. 350) So seven out of eight of these states were in effect created by the expansion of slavery in the antebellum United States, and Connecticut, with its heavily industrialized economy, had wealth that depended on the products of American slavery. Slavery was enormously profitable to slaveholders, though they often lived and died in massive amounts of debt. They tended to invest their profits in more assets - more land and more enslaved workers, which proved disastrous in financial panics and crashes like that of 1837, but large slaveholders were some of the wealthiest people in North America in the years before the American Civil War.

There is an old belief in our public understanding of American Slavery and the causes of the Civil War, a belief that had its genesis in the rhetoric of antislavery activists in the antebellum United States, that American slavery was not just a brutal, immoral, backwards practice, but an unprofitable one. They needed to believe this in part because their rhetoric depended so heavily on the conviction that free labor was not just morally superior but economically superior to slave labor. But when it came to the production of cotton, the 19th century world's most in-demand commodity, slavery was one of the most profitable investments a person, north or south, could make.

For more reading:

The Half Has Never Been Told: Slavery and the Making of American Capitalism by Edward Baptist

River of Dark Dreams: Slavery and Empire in the Cotton Kingdom by Walter Johnson

Battle Cry of Freedom: The Civil War Era by James McPherson

Empire of Cotton: A Global History by Sven Beckert