I've been a long time lurker and non-commenter. This post seems to not be getting much attention, so I'll throw my hat into the ring. I've been reading a book that touches on this subject--the way in which fascist regimes actually attained power.
My source is Robert Paxton's "The Anatomy of Fascism". This book seeks to uncover how "fascist" regimes were created and how they actually preformed. Some, like Germany and Italy, rose to prominence in ways that dominated their respective countries; some, such as France's and the UK's, failed completely.
But to answer the question. There are a variety of reasons that the NSDAP lost seats in the November 1932 election. The main reasons are:
These factors contributed to a loss in Nazi voting amounts.
1 is an interesting factor that we'll get to shortly, however it is fundamentally a perception/public relations problem. 2 and 3 are factors that affect the opposition and were always latent factors affecting Nazi numbers. Conservatives worked with the Nazis when they thought they would be fine; i.e. protect their traditional rights. Communists/socialist numbers attempted to oppose the Nazis, but were flat out unable to fight their violence and numbers.
Communist/socialist involvement with the Nazis was either political opposition or opportunist: they would utilize Nazi numbers when they thought the mutual benefits of striking together would aid in worker demonstrations. There is a well-documented account of German communists striking alongside Nazis. These co-operations were not unknown to the German electorate. However the German Left were still mostly against the NSDAP.
One must remember where the communist/socialists were coming from. Read, specifically, this passage from Paxton:
"Hitler thus built Nazism by July 1932 into the first catch-all party in German history and the largest party so far seen there. His Storm Troopers aroused both fear and admiration by their readiness to beat up socialists, communists, pacifists, and foreigners. Direct action and electioneering were complementary, not contradictory, tactics. Violence—selective violence against “anti-national" enemies who were perceived by many Germans as outside the fold—helped win the votes that allowed Hitler to pretend that he was working for power by legal means."
So the Nazis--already fascist and a faction willing to use violence to any end--were very aggressive physically previous to their greatest electoral victory. And, it seems, their tactics worked to the Nazis' benefit. In early 1932, the NSDAP seemed like a party that one could join to affect change in a specific way: if you wanted to kick the shit out of communists, you'd should a Nazi.
The socialist/communists of the time were not ignorant of the power machinations of the Nazis. They were convinced the Nazis would attempt a coup d'etat (which technically they never did). One can argue that their interpretation of fascism as violent is because, according to certain communist ideology, Nazis/fascists are essentially capitalist. That argument, however, is still up for debate.***
It should be noted that while the Nazis never gained a specific majority, they were still a force that conservatives in Germany were willing to cooperate with. Paxton maintains that fascism gained a foothold because of both the historical concerns of conservatives regarding loss of status, wealth, and rights as well as the development of mass culture and mass communication methods (not to mention a huge number of disaffected young men from WWI).
But, again, to get to the heart of your question, why did the NSDAP lose seats? The answer seems to be simple: The loss of momentum (1) within the movement:
"In November 1932, the Nazi vote slipped in further parliamentary elections. The Nazi Party was losing its most precious asset: momentum. Money was running out. Hitler, gambling all or nothing on the position of chancellor, refused all lesser offers to become vice-chancellor in a coalition government. The Nazi rank and file grew restive as the chances for jobs and places seemed to be slipping away. Gregor Strasser, head of the party organization and a leader of the movement’s anti-capitalist wing, was expelled for independent negotiations with the new chancellor, General Kurt von Schleicher. The movement might have ended as a footnote to history had it not been saved in the opening days of 1933 by conservative politicians who wanted to pilfer its following and use its political muscle for their own purposes. The specific path by which the fascists arrived in power in both Italy and Germany is the subject of the next chapter. But not until we have examined a third case, the failure of fascism in France."
The loss of momentum was completely tied with the Hitler's decision to choose to campaign for a powerful position within the Weimar government.
Hitler, like Mussolini, made a grand gamble for power. Whether or not Hitler and Mussolini knew that their opponents were weak, and therefore would ultimately "fold", is a question to a true historian. From what I can see, Hitler made a hard gamble for the German Chancellorship position and he managed to get it, despite his slight electoral loss with that recent election.
I think it's important to state this fact, because occasionally it is misstated: No Fascist movement managed to gain power through a violent coup or an electoral victory. Fascists worked within existing parties and bureaucracies--emulating and replacing those structures as necessary--to effectively seize power. They were elected/appointed, and therein then seized the power of government.
I hope this is thorough enough. I'd be happy to answer any questions.
***One argument based on Paxton's anatomy of fascism is that fascism doesn't exist on a deterministic course. Fascism existed a certain way historically, but that doesn't preclude it from a leftist leaning. To say: fascism doesn't have to be what it was, but it uses the methods that it does.