In his long review of the book the Party Decides and its application to the current election cycle, Nate Silver talks a little bit about the origin of the modern primary election cycle. The cases the book looks at are the ten competitive nominations between 1980 and 2004, but Silver criticizes the book for bracketing the weird cases in the 1970's, namely McGovern and Carter's runs as both managed secure the democratic nomination despite being factionally supported candidates ("My main objection is in how the authors tend to treat 1972 and 1976 as transitional cases that aren’t all that relevant to the current party system.").
The new system is apparently dependent on the recommendations of the McGovern-Fraser commission, which was formed in the wake of the imfamously tumultuous 1968 Democratic Convention in Chicago. The Wikipedia says:
One of the unintended consequences of McGovern-Fraser reforms was an enormous surge in the number of state party presidential primaries. Prior to the reforms, Democrats in two-thirds of the states used elite-run state conventions to choose convention delegates. In the post-reform era, over three-quarters of the states use primary elections to choose delegates, and over 80% of convention delegates are selected in these primaries. This is true for Republicans as well.
I was wondering if anyone could give more about this change, about how the nominating contests started to take this format, and particularly how Republicans ended up moving almost in tandem with the Democrats (though the first competitive Republican nomination was held in 1980, as the Republican nominees in both 72 and 76 were incumbents). How was this change an unintended consquence? Why did this system ended up getting chosen? I'm just looking for more history on the origins of the contemporary U.S. presidential candidate nominating process in general.
Excellent question!!!
At first, state legislatures elected electoral college officials. This was the case for the first few presidential elections, but went the way of the dodo about the same time as the First Party System.
In 1824, one of the most hotly contested presidential elections took place. Andrew Jackson won the popular vote and the majority of state, however, John Q. Adams, son of former President Adams, won the election. This was decided in the House due to no one getting a majority in the electoral college. This was Adams' and Clay's "Corrupt Bargain", in which in exchange for being named Secretary of State (seen as a stepping stone to the Presedency), Clay throw his support behind Adams. This lead to allegations that Adams stole the election.
This contested election happened in 1824 because the Democratic-Republican party could not get it together. The party broke along regional lines and instead of backing the official candidate, Sec. William H. Crawford, the some elements supported Adams, Jackson, or even Henry Clay.
At this time, Congressmen would get together, along with other important party members would chose the nominee at the convention. That is where the term brokered conventions would take place. These hotly contested affairs would routinely breakdown into heated floor debates.
Finally, Progressives in many western states started demanding the voters get to chose the presidential party nominees. This started in the late 19th and early 20th century. This slowly gain favor, but the nominee choice was still squarely in the hands of party officials. Think of a smoke filled backroom with the important players in a party hashing out who they can agree on to be the nominee.
Finally, 1968 rolls around. The 1968 Democratic Nomination Convention was a fun place. Mass riots. Tear gas. "The Whole World's Watching. This was all plastered over the background of Vietnam, the Tet Offensive, and in the early part of the year, Pres. Johnson surprising the world by not running for re-election.
This was contrasted with the smooth Republican nomination of Nixon and then the resultant blowout by the Republicans (Nixon won 32 states, Humphrey 13, and third party Wallace won 5).
This lead to, in 1972, a series of caucuses and primaries being implemented, so that the voters choose the nominee before the convention. This prevents another disastrous 1968-like convention. The Republicans quickly adopted the same idea.
Thus, in 1972, we say the modern primary process began.
Here is a source. Not the greatest, but the only one in front of me. If need be, I can find a book or two later to recommend.
I don't want to stray beyond 1972, as the answers there get complicated and may blur the line with the 20 year rule.
The reason why many states shifted to Presidential Primaries following the McGovern-Fraser Commission's report can essentially be boiled down to two words: regulatory uncertainty.
Prior to 1972, state delegates were chiefly chosen through one of four different methods: a party caucus in which only certain people can participate; an open caucus (the ones we have today, like the Iowa caucuses); a delegate primary in which voters elect the delegates to go to the convention; and a direct primary (the ones we have today). In addition to those four, you also had the non-binding "beauty pageant" primary in which voters voted for their choice of nominee, but it was non-binding and the state party could ignore it.
The McGovern-Fraser Commission banned all but two of those systems: the open caucus and the direct primary. In doing this though, it also established a very long list of regulations for how states had to appoint their delegates to the Democratic convention. Failure to comply would mean failure to have your delegates seated at the convention.
Thus, rather than risk failing to comply (regulatory uncertainty), most states opted for the easier to implement of the two systems: the direct primary.
Regarding the Republicans, they were caught up in this due to the spillover effect. When states were rewriting their laws to comply with the new rules for the Democratic party, they usually wrote it in such a way that it would apply to both parties.
It's important to note though that the Republicans did have their own voter reform commission around the same time called the Committee on Delegates and Organization (DO). Much like the McGovern-Fraser Commission, its aim was to increase voter participation and broaden representation, but the Republicans didn't feel as compelled to make changes as the Democrats did. Their party was more unified at the time, and the 1964 nomination of Goldwater already showed that grassroots activists could work well within the existing framework. Thus, the DO report was much less expansive, focusing more on ensuring delegates represented women, minorities, and youth.
Moreover, the McGovern-Fraser Commission was far from the final attempt by the Democrats to reform their nomination selection rules. There'd be 4 more after it, included the Hunt Commission in the 80s that added the superdelegates rule. But, of all the voter reform reports, it was by far the most influential.