Hello there! Here is a .pdf containing the full version of the poem. The portion you have in bold is one of the references to slavery, as well as the following:

"O thus be it ever when freemen shall stand."

Outside of writing the Star Spangled Banner, Key was one of the founders of the American Colonization Society (ACS), an organisation whose goal was to aid free blacks that wanted to go to Africa. Other founders include individuals such as James Monroe and Andrew Jackson. both of which owned slaves. There were also abolitionists such as Henry Clay, author of the Missouri Compromise, that supported the ACS. People like Frederick Douglass would later be vehemently opposed to colonization, saying:

"In view of this proposition, we would respectfully suggest to the assembled wisdom of the nation, that it might be well to ascertain the number of free colored people who will be likely to need the assistance of government to help them out of this country to Liberia, or elsewhere, beyond the limits of these United States—since this course might save any embarrassment which would result from an appropriation more than commensurate to the numbers who might be disposed to leave this, our own country, for one we know not of. We are of the opinion that the free colored people generally mean to live in America, and not in Africa; and to appropriate a large sum for our removal, would merely be a waste of the public money. We do not mean to go to Liberia. Our minds are made up to live here if we can, or die here if we must; so every attempt to remove us will be, as it ought to be, labor lost. Here we are, and here we shall remain. While our brethren are in bondage on these shores, it is idle to think of inducing any considerable number of the free colored people to quit this for a foreign land."

Variations of this plan have been advocated by a wide spectrum of people including Marcus Garvey, Malcolm X, and the KKK.

Whether or not he supported the perpetuation of slavery as a practice is not explicitly discussed in any of the letters that I have read by him. The ACS is, by contemporary standards, inherently racist and essentially the precursor to segregation (By acknowledging that whites and blacks were best not to mix). Using contemporary standards to evaluate historical figures is troublesome as many of these people believed that they were acting in the best interests of slaves. Key came from a slaveholding family and owned several himself, yet is compared to people like Jefferson as a gentle slave owner. Encyclopedia Brittanica claims he was also an early and ardent proponent for ending the slave trade, though I could not find anything (in my brief search) corroborating this.

For example, Key writes:

"Dr Ayres says he is making preparations for a school. He means to adopt in some measure the plan of Mr. Johnson at Regent’s Town, & keep the captured Africans & the boys together. - He has put them for the present under the charge of Lott Carey. - The young man you speak of will be eminently useful, and I trust the Lord will preserve him."

Gauging the benevolence of someone's actions is based on opinion, so you would have to declare for yourself whether you think his actions reflect someone that is racist. You will not find a historiographic consensus on the subject, especially with how political the debate has become recently.

Biographical information from Gilder Lehrman Institute of American History, Encyclopedia Brittanica sections on Francis Scott Key, Henry Clay, and the American Colonization Society,