From what I understand, some of the important factors were Dixiecrats finally becoming Republicans, the Contract with America, and inefficiency during Clinton's first years, but it doesn't seem like Clinton did anything that unpopular to begin his Presidency. So why exactly did his party get crushed?
The entire 'Hillary Care' push by the Clinton adminstration was a mobilizing issue for the Republicans in 1994 similar to what we saw with ObamaCare was in 2012.
Add in the issues that you bring up and we saw a big Congressional shift.
We also saw a shift in Congress, albeit less dramatic, in the 1950 & 1952 elections, which followed Truman's proposal for universal health care as part of the Fair Deal.
In those two election, Republicans gained 50 seats and took control of the House in 1952.