Although their careers overlapped James Brown did not make his debut on the Ed Sullivan show until 1966, while the Elvis shows were in 1956. In the intervening ten years television censors had become more relaxed plus the US had undergone the Beatlemania phenomenon in the mean time.

Secondly, Elvis's dancing style had significant hip motion while James's dancing had mostly knees and feet which conforms more with traditional notions of dance propriety.

A follow up question: was there much of an intersection of Elvis and James Brown fans / subcultures?

My presumption is that Elvis attracted predominantly white audiences while James Brown attracted predominantly black audiences, but I'm hoping someone who knows more in this can let me know if that presumption is sound or not.