Rockwell was actually a fan of Malcolm X because Malcolm X supported (at the time at least) a separation of the races. Both felt that there could be no lasting peace or stability unless the whites and blacks were physically separated in different countries.

Rockwell referred to the Nation of Islam as "Black Nazis" who shared complementary ideologies in terms of their extreme distaste for all things involving race-mixing.

Edit for correction : Additionally, Rockwell and his ANP went to more than 1 Nation of Islam rally in the early 1960s and Rockwell even spoke following Malcolm X Elijah Muhammad at one of them. Rockwell paid Malcolm X Elijah Muhammad the most supreme "compliment" calling Malcolm X Elijah Muhammad the "Adolf Hitler of the black man"

I will write up a more thorough comment when I'm not at work and have access to my research and citations.

Source: Wrote a thesis on the American Nazi Party but for one book in particular, check out Hate by William Schmaltz. It's an easier read but pretty comprehensive.