So most of this argument (on both sides) has occurred within the last 20 years, and so will be difficult to discuss given the rules of this sub. A couple of points, however, might help to elucidate your question:

1. The lyrics of the third verse, very clearly, discuss the usage by the British of hired soldiers and (now ex-) slaves to fight against the Americans, and Key is clearly railing against this, as could be expected when a foreign army invades your country and incites your slave population to fight against you. The lyrics do not openly take a position on slavery as an institution, although the repeated usage of "land of the free" alongside a clear acknowledgment of the existence of slavery in said "land of the free" does suggest that the author did not see, for whatever reason, the mere existence of slavery as being problematic when calling the US the "land of the free".

2. Key was a slave-owner and a lawyer who actively defended slave holding interests, so it's by no means a far stretch to read his writing and see a pro-slavery stance where a more objective interpretation that ignores the authors otherwise well-known pro-slavery beliefs might see less support for such a stance.

This National Review article - a conservative outlet - sums up the argument pretty well by arguing that, yeah, it's probably racist and pro-slavery, but then so is just about everything else from this era of American history (and much later too), so if folks want to ditch it for something else, they are going to have to really search for a suitable replacement.

I'm particularly curious now because @Trillburne has a thread on Twitter claiming that the line is not about slavery, but is instead referring to the American belief that the only way a soldier could fight for a king is money ("hireling") or coercion ("slave"). His argument seems convincing but the historical consensus seems to be the opposite?