I am aware of no such controversy. The Unitarian Church of the nineteenth century is the spiritual predecessor of today's Unitarian Universalist Association, but there are significant differences. The American Unitarians emerged in the late eighteenth and early nineteenth centuries as an outgrowth of a liberalizing tendency in the Congregational Church (the direct descendent of the Puritanism of 17th century Boston). By the early nineteenth century Unitarians had captured Harvard Divinity School, and thus most of the clergy of New England as well as its premier educational institution. Unitarianism became the official religion of educated New England, including the major figures of the Transcendentalist movement. Notably, Ralph Waldo Emerson was trained at Harvard as a Unitarian minister, eventually leaving the ministry but indelibly shaped by Unitarian thought. The fact that both Adams presidents were Unitarian would not only not have been a surprise, but it would have been shocking if these two Harvard-educated Boston literary and political elites had not been Unitarian.

I'm not as familiar with Fillmore, but he was from Western New York, and appears to have come to Unitarianism later in life. His cabinet consisted mostly of Southerners and Westerners, but Edward Everett, one of his Secretaries of State, was Unitarian.

I didn't know Taft was Unitarian, to be honest.

Finally, the American Unitarian Association merged with the Universalist Church of America in 1961. The resulting entity uses the name Unitarian, but does not actually espouse unitarian theology any longer, preferring a less dogmatic and more eclectic creed that emphasizes personal choice in religious belief. It is these "Unitarians" who many people later in the twentieth century (twenty-year rule!) made fun of. In particular I remember growing up and seeing a lot of jokes about my religion on The Simpsons. According to The Toast, Matt Groening may have been Unitarian. The religion is pretty liberal in its theology, and is no longer even technically Christian. It is these Unitarians who might have been expected to have been subjected to an uphill battle for the Presidency. In the nineteenth and early twentieth centuries Unitarianism was the main religion of one of the country's most culturally, socially, and financially powerful groups.

Source:

Lawrence Buell's Emerson is a good accounting of one of America's most important intellectuals that includes quite a bit on the situation of Unitarianism in the middle of the 19th century.

Dorrien, The Making of American Liberal Theology is the standard work on, well, American liberal theology.