By May of 1980 (this point 36 years ago) it was clear Carter was in serious trouble. The humiliating failure of the attempted rescue of Iranian hostages in April 1980 coupled with the economic crisis and the feeble response to the Soviet invasion of Afghanistan combined to make Carter seem like an incompetent bungler. Reagan's hawkish rhetoric and right wing politics had led liberal Republican Congressman John Anderson, one of Reagan's primary rivals, to launch an independent bid, which Carter hoped would split the GOP. In the end, Anderson's support came heavily from Carter.

Additionally, the Democratic Party was disorganized and bitterly divided by Teddy Kennedy's primary challenge to Carter, which Kennedy kept up until the convention, trying to get delegates unpledged in an attempt to win on the convention floor. This divide left the left wing of the party, who had backed Kennedy, bitter and unenthusiastic about Carter.

There were indeed doubts about Reagan's electability in 1980. I don't feel that I can comment in detail about how widespread they were, but they pop up in news media from the time and in history books.

For example here is an NBC piece on the question of if Reagan can win: https://static.nbclearn.com/files/higheredsa/site/pdf/3252.pdf

Here is an AP piece quoting Former President Ford saying that Reagan couldn't win: https://news.google.com/newspapers?nid=1696&dat=19800303&id=xHQ1AAAAIBAJ&sjid=_kYEAAAAIBAJ&pg=6656,269794&hl=en

The concerns about Reagan's electability were centered in the question of if Reagan was too conservative for the country. To understand why that would be a concern, we need to take a step back and look at the world of 1980.

From 1948 to 1980 there had been six elected presidents (Ford was never elected and only served out Nixon's second term). Four of the six were Democrats (Truman, Kennedy, Johnson, Carter) and two were Republicans (Eisenhower and Nixon). All of them governed in what one might call the New Deal consensus.

That is to say they all supported efforts to build up the nation at home through federal government policies. Even Nixon, who often dreamed of undoing much of the progress of LBJ's Great Society, was basically content to operate within the New Deal consensus framework.

Ronald Reagan, who famously said things like ["The nine most terrifying words in the English language are: I'm from the government and I'm here to help,"] (https://www.youtube.com/watch?v=xhYJS80MgYA) was a clean break from that tradition. Instead Reagan looked a lot like Barry Goldwater, whose 1964 nomination led to the worst defeat the Republican Party suffered in the post-World War 2 era. This was enough to concern some and hearten others that Reagan would similarly be defeated.

To be specific beyond just questions about the size and role of government, Reagan was saying things that had historically been associated with extremist groups. For example his support of 'states rights' was an echo of segregationists opposition to federal action on civil rights issues.

Sources:

http://www.amazon.com/White-Protestant-Nation-American-Conservative/dp/0802144209 http://www.amazon.com/Before-Storm-Goldwater-Unmaking-Consensus/dp/1568584121 http://www.amazon.com/Presidential-Campaigns-George-Washington-Bush/dp/0195167163 http://www.amazon.com/One-Man-Against-World-Tragedy/dp/1627790837