This is an interesting question. As you observe, for much of the nation's history, Roman Catholicism was heavily feared and vilified by Protestants in the United States, who were hostile to the idea of Catholics holding higher office. Catholicism became much more acceptable in the aftermath of the second world war. Some of the reasons for this have to do with changing attitudes about religion in society, but it also has to do with changes to Catholicism that made it seem less threatening to Protestants.

Before the Twentieth Century

Protestant hostility towards Roman Catholics did not mean they were unknown in government prior to the twentieth century. Charles Carroll is famous as the only Roman Catholic signer of the Declaration of Independence and was elected in 1788 as the U.S. senator from Maryland. Roger Taney, the infamous chief justice of the Supreme Court who decided the Dred Scott case, was a Catholic. Before becoming Chief Justice in 1836, Taney had been Attorney General and Secretary of the Treasury.

Protestants, however, were extremely scared of Catholics in political power. The ratification of the Constitution's provision barring religious tests in public office was controversial in part because of fears of the possibility of a Roman Catholic becoming president. There were even hypothetical arguments advanced in the ratification convention about the idea of the Pope becoming president.

Part of the concern of Protestants was that Roman Catholicism was seen as hostile to republican governance. They saw the Pope as a foreign monarch, and Roman Catholics as in submission to his authority, which they worried overrode their loyalty to the United States. There was also an element of racial and ethnic prejudice, as Roman Catholic immigrants tended to be Irish, and later Italian, rather than from England, France, or Germany.

The Roman Catholic church also opposed the separation of church and state. In 1864, Pope Pius IX issued the Syllabus of Errors, which declared it was an error that “That the Church ought to be separated from the State, and the State from the Church.” This stoked Protestant fears that Catholics would institute religious control of the country if they gained political power.

Al Smith and the 1928 Election

In 1928, Al Smith, a Catholic governor of New York, ran as the Democratic candidate for president against Republican Herbert Hoover. Smith massively lost the popular vote and electoral college. Though Hoover was popular in his own right, anti-Catholicism was a huge factor. Smith had trouble competing even in the South, where Democrats had traditionally dominated since the Civil War.

Part of the issue was that Smith was opposed to prohibition; a position typically called being a “wet.” Catholics, particularly Irish and Italian Catholics, were often hostile to prohibition, which they saw as an attack on their cultural mores. Hoover’s support for prohibition meant that people could support him without ever directly making their votes about religion.

That said, many people were overtly hostile to Smith because of his faith. The Ku Klux Klan apparently had real concerns in the election, and had trouble deciding between a Republican (which they opposed due to the Civil War) or a Catholic. Many Protestant clergymen in the South refused to vote for Smith.

Teaching Tolerance

There was a major push after World War I to emphasize tolerance and pluralism. In 1927, the National Conference of Jews and Christians (Protestant and Catholic) was set up. This would become the National Conference of Christians and Jews (NCCJ) in 1938.

The NCCJ promoted the idea of interreligious cooperation. For example, the group sent out “tolerance trios” of a Protestant minister, rabbi, and Catholic priest to travel the country and give talks about how they all supported each other. Such efforts were controversial with conservatives in all three religious groups, but proved effective.

In the led up to World War II, there was what can either be termed an education or propaganda effort by the U.S. government to promote religious cooperation. The tolerance trios toured military camps. The government promoted “Brotherhood Week,” emphasizing the connection between Catholics, Protestants, and Jews.

One of the more interesting propaganda efforts centered around the story of the “Four Chaplains.” These were four chaplains, two Protestants, a Catholic priest and a rabbi, who were on the troop ship USS Dorchester when it was torpedoed by a U-boat in 1943. The ship did not have enough life vests, so the chaplains helped as many soldiers as they could, heroically gave up their own vests, and prayed together as the ship sank and they drowned. The government promoted this story as an example of interfaith cooperation.

After the war, interfaith cooperation began to center around the idea of a shared “Judeo-Christian tradition.” As the Cold War began, this came to be seen as a counter to atheistic communism. Catholicism was no longer frightening to Protestants; rather, along with Judaism, it could be patriotic.

Note: Too long, continued in my reply below.