You're on the right track. Ellis wrote a chapter about slavery which had a lot of moving parts and he wasn't quite as clear about them as he could have been.

There are two important parts here, one of which is reasonably easy to tease out and the other being quite murky. So let's start with the easy part: The Constitution prohibits any ban on the import of enslaved people from Africa or other points abroad until 1808. Until that time, Congress is forbidden to do any such thing. Afterwards, the option exists. It's take at the first available instant, the ban becoming legally (but not substantially) effective at 12:01 AM, 1 January 1808. Especially in the early years there is a lot of smuggling of people, including a notorious case where the former governor of Georgia was part of an illegal importation scheme that involved using a member of the Creek Nation as a potential fall guy and bringing people in through Spanish Florida, for transit through Creek territory and thence out to as far west as Texas where one of the financiers shows up as an early-ish American colonist. But enforcement does improve over time and smuggling of enslaved people falls of significantly around 1820.

That is the slave trade, but only in part. Any time one is buying and selling enslaved people one is engaged in the slave trade. Nothing in the laws above prohibited sale of humans between Americans within the bounds of the US. Indeed, even if one got caught breaking the ban on importation it meant you got fined but the people you enslaved remained enslaved and would be sold at auction down the road. Technically the matter was left to the states where the people were discovered and prosecution took place, but in practice that's always going to be an enslaving state and they are not in the general manumission business.

What Ellis is mainly talking about there is not the ban on importation or the slave trade general, except insofar as abolishing slavery would mean the end of both. Specifically the petitioners, including Ben Franklin, asked that Congress do something about slavery with an eye toward its eventual extinction. Assuming a majority for that existed (which, I should add, is a bad assumption for any congress until 1861 at the earliest) the next question would be how far Congress' powers reached? Could it pass a law requiring enslavers sell their enslaved people to the nation at a fixed rate, at which point the nation would then manumit them and (probably, because of vicious white supremacy) deport them back to Africa or to somewhere in the American West that white Americans choose to view as empty because no one they acknowledge as worthy of consideration, except genocidally, lives there? There's neither an explicit authorization for such a plan nor an explicit prohibition. What does that mean?

All texts, whether laws or otherwise, are perpetually under social construction. The words to not speak for themselves, but rather signify meanings that people frequently contest. The Constitution is no different, for all that some will try to claim that their preferred reading is simply the null interpretation and everyone else is doing politics. Actual "Constitutional" principles are essentially the speaker's policy preferences of the moment, which Americans for and against enslaving people demonstrate consistently for as long as slavery exists. So the argument for or against Congress' power to emancipate is couched in constitutional terms of what is allowed, it's really about which outcome a majority of Congress prefers. It is, lamentably and unsurprisingly, the horrifying one.

You can't just ignore a petition signed by the ailing Franklin and differences over slavery are so explosive that they disrupt the allegedly great business of the day regarding Alexander Hamilton's financial plan. This is about the pattern for slavery arguments: every other great issue, no matter how imperative, is a relative sideshow. Because the differences are stark and intractable, Congress decides to declare its hands tied on the matter. The report out of committee declares that Congress lacks constitutional power to do anything at all about slavery within the bounds of a state. That protects slavery everywhere that actually has representation, though it leaves an open question as to its statue in the territories. That in due course becomes the front on which the remainder of the question will be fought until 1863.

This is a capitulation presented as a compromise, which makes it marginally more acceptable to antislavery Americans and swiftly becomes the received wisdom of both white parties. Hence Ellis quotes Daniel Webster, decades down the line from 1790, declaring that no other notion had ever been entertained. Lincoln would argue the same, declaring that neither he nor his avowedly antislavery party intended to alter the situation within the enslaving states. That's the Silence to which Ellis' chapter speaks, though it turns out to not be remotely silent. Insofar as slavery is not debated, it is because powerful white men of both sections decide that antislavery sentiment must be contained and frustrated and they devote considerable energy to that project. Declaring Congress so powerless against slavery was a choice consciously made in just that vein.