Largely they weren't classified.

I've been digging through some primary sources now for a couple hours, and I haven't seen them "grouped" like that with anyone. Instead they seem to be primarily referred to in terms of blasphemy and heresy. Take for example The Antidote an 1857 anti-Mormon publication. It's full title was "The Antidote to Mormonism and Infidel Error: Exposing the falsehoods and frauds and replying to the blasphemous pretensions of Mormonism; refuting the infidel arguments and 'vain philosophy' of 'secularism' and providing sound and concise arguments in support of divine truth." Clearly then the authors of The Antidote saw the Mormons as something akin to a Christian error. Their tone is consistently one of correcting what they appear to view as a heresy within Christianity. The Reverend MT Lamb, in his 1887 book The Golden Bible, or The Book of Mormon, Is It From God? seems to take the same tact. That it's a Christian heresy. He doesn't bother classify it as any kind of separate religion. That pattern is repeated in anti-Mormon tracts through the 19th century. LDS church critics seem to largely view it from within a Christian viewpoint. They use that to condemn it as blasphemous. What they don't seem to do is treat it as outside Christian, like they do Islam, although they love to make comparison between Smith and Mohammad

Of course that only tells us what the contemporary academics and critics thought. I've got a few firsthand accounts that seem to largely agreee with the critics. It wasn't quite viewed as proper Christianity, because it was wrong (in their minds). But it doesn't seem to be expressed as a brand new category of belief either. Basically no one at the time seems to have much considered your question.

All of the old "Censuses of Religious Bodies", done by the Census Bureau before it was forbidden to ask questions about religion, are available as PDFs online. Here's the one from 1906. Almost all the 145 religious bodies are Protestant but you can see on page 17 they don't count 22 of these bodies as Protestant. They put the non-Protestant bodies in five groups:

1. the Roman Catholic Church

2. Jewish congregations

3. the Latter-day Saints (consisting of what we today think of the Mormon Church and the Community of Christ)

4. Eastern Orthodox Churches (Russian, "Servian", Syriac, and Greek—today we'd recognize the Syriac church as Oriental Orthodox rather than Eastern Orthodox).

5. All other bodies, which is a real grab bag and includes every one from Armenians and Shakes to Buddhists and Bahais to Theosophists and the Sociey for Ethical Culture.

It's interesting to see here that the LDS churches are recognized as neither fish nor foul; they don't fit into the main categories (Protestant, Catholic, Jewish, Orthodox), but they're also not relegated to the grab bag category. They're recognized as their own thing. While this report is from 1906, I get the impression this is true for much of the later 19th century, though I haven't tried looking through the earlier surveys (1906 was the first of the new format, but I believe various census surveys of religious bodies go back to 1850).