This is actually an extraordinarily complex question, one that I can't really give you a thorough answer to without addressing some broader theoretical problems and wider movements in the scholarship of unfree labour. I'm going to try and do that as succinctly as I can.

In a nutshell: as a theoretical system, there is no reason why the State must exist for slavery to be viable as a profit-making enterprise. In practice however, the two are so intimately tied up with one another that you can't really speculate much about what would happen to slavery as a system in the absence of the State, beyond that it can thrive. In practical terms, the State in the antebellum USA both legitimised and dismantled systems of oppression that characterised racial slavery, giving it a mixed role.

Slavery as it was practised in North America is one of many forms of unfree labour that have existed in the world. One of the broader theoretical implication of your question, then, is do extreme forms of unfree labour depend on government support to be economically and commercially viable? The basic answer is no - at least, not in the way you imply in your question. For evidence of that fact we need only look to the present day; the International Labour Organisation estimates that there are 21million victims of unfree labour, including slavery-like practices, today. Of those, some 90% are victims of illegal exploitation by private enterprise or individuals, and they generate billions and billions of dollars in global profit. So yes, unfree labour can be profitable without government sanction or direct support.

In truth though, it gets more complicated when we start to pick apart your question. Your question rightly identifies that the State had a very profound role to play in facilitating slavery in the US, though it certainly didn't socialise it in the sense that the word is commonly used in American political discourse today. That would imply the spreading of a burden of public subsidy across the community for its collective benefit with an implication of common ownership; this is fundamentally and dramatically different to how the State facilitated and supported slavery. Supporting slavery may have benefited the wider economy, but it was certainly not for the purposes of benefiting the wider community of citizens at a time when governments were concerned chiefly with the well-being of a narrow class of educated, affluent voters.

But slavery did not emerge from the State; it was not created one day by the government with all of the facilities it needed to succeed. As I touched on in this old post, in the early history of the British colonies, it appears that it actually took some time for the legal and administrative theoretical conception of slavery to emerge, even after the social reality experienced every day on the plantations had existed for some time. Neither the US nor British governments were pro-active architects in building slavery in the way your question implies but they grew to become eager facilitators of it; they created legal and institutional frameworks that made it easier for slave owners to enforce their existing authority over their slaves. By the time many of the more rigid limitations imposed on slaves had been codified into law, they had been widespread practice already; certainly slavery had been profitable for a long time before many of the more infamous provisions of the slave codes found their way onto the statute books.

Your question also presumes that these things were the exclusive domain of the State. Certainly in the United States, State and federal governments came to play a prominent part in the crushing of resistance or independence. But they were far from the only agents responsible for doing so - in fact, the State ultimately had quite a small part to play. As I explained in this answer from many months ago, the entire system of racial slavery - the entire process by which one became a slave - was constructed to minimise and stamp out any serious ambition or possibility of resistance before a slave had even reached their ultimate destination. When resistance did arise, it was generally dealt with with comparatively limited government involvement; slave patrols were formed by local white men long before the United States even existed. Whilst state intervention certainly made the process of dealing with escapees easier, it certainly wasn't essential. The vast majority of violent repression was carried out by private slave owners on private property.

With regards to large-scale revolt, repression of these represented both the State repressing anti-slavery movements as well as fulfilling its obligation to the citizenship to provide for public order and safety. Today, most of us would expect our governments to put a stop to violent rioting endangering property and life, even if we understood why people were rioting and felt sympathetic to them. In the same way, the authorities of the day were obligated to uphold law and order in the face of the kind of violent chaos a violent slave revolt could possibly herald. It is thus quite erroneous to assume that even if every branch of government had said "sorry, we're not doing anything to help you enforce slavery, we're just going to tolerate it" that it would have not stepped in to stop an open revolt.

The most crucial weakness in your question though is the assumption that the government 'socialised' the cost of capturing slaves. In reality, the US government played a key role in simultaneously destroying and reinventing the slave trade. Until the revolutionary war, slaves had generally arrived on British ships from Africa or the Caribbean; those arriving on American ships generally had ties to British finance or slave capturers. Thus for the duration of the revolutionary war, trading in slaves was banned by the rebelling colonies; though a few states tried to re-open the trade after independence, by 1794 the transatlantic trade was banned - though the Constitution explicitly said it could not be made illegal until 1807 - in totality across the young USA. From 1807 onwards the trade was illegal proper at the federal level. Despite this, we know for certain that some 80,000 new slaves were imported to the USA between 1800 and 1820, and suspect many more went unaccounted for. Rather than the slave trade being facilitated by too much government support, it persisted arguably in spite of government repression - clearly, it was still profitable. At the same time a vast internal trade in slaves, taking advantage of the fact that some states had a constant surplus of slaves and others a constantly growing need to fuel their growing plantation economies, developed within the USA. When faced with the government demolishing the infrastructure of slavery, slave society merely turned to an alternative way to keep the trade alive.

The State then was simutlaneously an an instigator, facilitator and repressor of slavery - all at once. This is the harsh reality of the nature of the State's relationship with unfree labour: whilst we like to think of it as a very black and white issue, government policy does not fit easily into either being 'pro-slavery' or 'anti-slavery'. This remains true to this day - today, the United States decrees slavery to be abhorrent, and the trafficking of people across borders for exploitation a crime against Humanity. At the same time however, it makes use of prison labour, and many scholars argue Human trafficking is only possible because people seek to subvert restrictive immigration policies imposed by the government. Thus today, the State is simultaneously a repressor, instigator and facilitator of unfree labour; the same was true of the State during the time of slavery.

For this reason, to turn to your specific question about profitability, an answer is difficult to give because we simply cannot remove the State from the equation. This is because of the very nature of slavery - or rather, freedom. In the modern western world we are very accustomed to thinking of freedom as a virtue we define in positive terms. You are free to do this; I am free to do that. But the reality is that isn't what freedom is about. We define our freedom in terms of the things that other people cannot do; freedom is a relationship between two or more individuals, not an absolute principle. Ownership is an expression of this relationship. If you own something, culturally we interpret that as a freedom; you are free to do what you want with that box. In practice though, that doesn't mean much. Physically I can come along and pick your box up just as you do. Socially however, I am expected to respect the fact that it is your box. Your freedom to do what you want with your property means that I cannot do what I want with it, and if I try to, you can take action to prevent or punish me.

So slavery is an extension of the social construct of freedom. It cannot exist in nature but it can be constructed in a society by redefining the social relationship between slave, owner and non-owners. That in turn has implications for the law and State. Both these things are part of our social contract; they reflect our relationship with the community as a whole. For that reason, you simply cannot remove the State in totality from slavery; how the State treats slavery has profound implications for how the relationships that define slavery work. Put simply, it was simply not possible for the government to do nothing about slavery - because doing nothing still has implications for the social reality. So we cannot really explore the counter-factual your question seeks.