Was there ever a motion to essentially transition from slavery to something like institutionalized serfdom in the antebellum period? A model where black people would not be slaves anymore, but not free either, and bound to the land of their former owners etc., similar to Russian serfs for example. Anything of that nature? I could imagine someone proposing this, to make the situation look better internationally and to appease the less fervent abolitionists, without changing that much for the planter elite.
You may be interested in this two-part answer of mine, which gives a rundown of all the schemes to abolish slavery in the South before the Civil War.
The short answer is no, there were no serious attempts to transition from slavery to serfdom. The scheme that got the most traction is the same scheme that most of the Northern states had used to abolish slavery: "gradual emancipation", sometimes known as a "free womb law".
The law would designate some date after which everybody would be born free. People born before that date could still be enslaved for life. Children born to enslaved mothers after the designated date would follow the condition of their mother until they reached the age of 18, or 21, or 25, or even 28, depending on how the law was written.
There was sometimes a later date, upon which everyone still enslaved would be freed, generally 25-50 years after the "free womb" date had been reached. Though sometimes there was no such date, so that the last enslaved person would conceivably die off some 80-100 years after the "free womb" law went into effect, having been born just before the cutoff date.
These "gradual emancipation" laws were often paired with "compensated emancipation" laws, which would mandate that the government pay some sort of monetary value to the slaveholders upon emancipation, for the "loss of property".
The furthest this ever went in the South was in Virginia in 1832. And it didn't go much of anywhere at all. It never got a vote, just a debate in the state legislature. What did get a vote during the debate was the proposition that emancipation/abolition could never be discussed openly by the state legislature again. That vote failed, but at least it got a vote.
A couple of other slave states discussed it in the antebellum period during debates while re-writing their state constitutions. But it didn't gain much sympathy there either. The new state constitutions in the South written in that period were generally more restrictive when it came to slavery, not less.