This is a really broad question.

Partly racism. Civil rights were seen as a huge threat to many whites. The very contentious Brown v. Board of Education was only about 12 years old when the Panthers were founded. Imagine being a racist white person who went from having black people not allowed to sit next to you on a bus to holding armed rallies in state capitals in a 15 year period. That is a drastic change. Armed black people were (and probably still are) a terribly frighting image to many.

Second, there was a diverse and influential New Left movement which had sprung up and was challenging many social norms. The fact that these challenges were taking place in the midst of the cold war was especially alarming. The Soviets were still seen as a huge threat. The Panthers were Marxist-Leninist-Maoists. Imagine being in the middle of the Cold War and having an armed communist group form inside the USA.

Also, it was the Panther rhetoric itself. They called police 'pigs' and loved posing with guns. They did have shoot outs with police and they did stand up to a brutal and racist police force (at least in Oakland, where the panthers were formed.) They were not the only militant group of the time, nor the only armed Black Power group, but they were able to get the most headlines. The media loved what they were doing. Armed black militants sold papers. Even later on when the Party largely moved away from armed confrontations and focused on things like community breakfast programs, they were unable to shed this militant image.

Finally, though they only had a small amount of members, they had a large following and influence that went well beyond their membership. In 1972, when the Party was in its decline, Bobby Seale ran for mayor of Oakland and got over 19% of the vote, forcing a runoff election. That is tens of thousands of people willing to vote for a member of the Party.