I'll answer the first part of the question: "What historically would make the anthem racist?" As you may be aware, the National Anthem comes from the poem "The Star Spangled Banner" originally written by Francis Scott Key as he observed the shelling of Fort McHenry by the British Navy during the Battle of Baltimore. Held as a hostage on the ship, he wrote the following poem, the first verse of which is the National Anthem. The third verse bears most of the explicitly racist lyrics, which I embolden below.

O say can you see, by the dawn's early light,
What so proudly we hailed at the twilight's last gleaming,
Whose broad stripes and bright stars through the perilous fight,
O'er the ramparts we watched, were so gallantly streaming?
And the rocket's red glare, the bombs bursting in air,
Gave proof through the night that our flag was still there;
O say does that star-spangled banner yet wave
O'er the land of the free and the home of the brave?

On the shore dimly seen through the mists of the deep,
Where the foe's haughty host in dread silence reposes,
What is that which the breeze, o'er the towering steep,
As it fitfully blows, half conceals, half discloses?
Now it catches the gleam of the morning's first beam,
In full glory reflected now shines in the stream:
'Tis the star-spangled banner, O long may it wave
O'er the land of the free and the home of the brave.

And where is that band who so vauntingly swore
That the havoc of war and the battle's confusion,
A home and a country, should leave us no more?
Their blood has washed out their foul footsteps' pollution.
No refuge could save the hireling and slave
From the terror of flight, or the gloom of the grave:
And the star-spangled banner in triumph doth wave,
O'er the land of the free and the home of the brave.

O thus be it ever, when freemen shall stand
Between their loved homes and the war's desolation.
Blest with vict'ry and peace, may the Heav'n rescued land
Praise the Power that hath made and preserved us a nation!
Then conquer we must, when our cause it is just,
And this be our motto: 'In God is our trust.'
And the star-spangled banner in triumph shall wave
O'er the land of the free and the home of the brave!

The part highlighted above deserves explanation. The first "Their" refers to U.S. Army and Militia troops specifically putting down poor white and enslaved peoples' insurrections that were egged on by the British, who offered reward, including freedom for those that did works of sabotage against the U.S. The second "their" refers to the "hireling and slave" of the following line. Key refers to them as "pollution," which suggests an inhumanity and odiousness that is very much tied to their class, race, and ostensible lack of patriotism. Key also promises no quarter to these folks, who will find "No refuge" from "terror" or "grave." That is, they deserve nothing but total slaughter. He concludes that stanza with the haunting image of the triumphal U.S. flag waving over a land purified of insurrectionist enslaved and malcontent "hireling[s]."

Since both groups--hirelings and slaves--are tied to race and class, with slavery being preserved especially and solely for people of African descent, the racial connotations are there in the unity of the poem's elements, even if they aren't necessarily explicit in the Anthem as sung in sports arenas.The context of its composition and Key's broader intent to threaten the "hireling and slave" color the way folks interpret the singled-out part that is sung.