This Time magazine article goes into more detail:-
Goldfinger took the cat from under his arm and tossed it to the Korean who caught it eagerly – “I am tired of seeing this animal around. You may have it for dinner.” The Korean’s eyes gleamed.
“They are the cruelest, most ruthless people in the world…When they want women, street women are brought down from London, well-remunerated for their services and sent back. The women are not much to look at, but they are white and that is all the Koreans ask – to submit the white race to the grossest indignities.”
Bond apparently share's Goldfinger's view:
Bond intended to stay alive on his own terms. Those terms included putting Odd-Job or any other Korean firmly in place, which in Bond’s estimation was lower than apes in the mammalian hierarchy.
It's a combination of Ian Fleming's own personal prejudices (as many and varied as they were) and Britain's imperialist nostalgia post-WWII. Let me clarify. For reference, I'll be drawing on direct quotes from Fleming's work, trends and events as they happened in the Late British Empire, as well John Pearson's biography of Fleming himself and an article in the New Statesmen by William Cook who was citing Pearson called "Novel man".
First, Fleming. Fleming was the son of a parliamentarian with family banking connections. He grew up during World War 1 and came into adulthood during the Great Depression. Britain was still an imperial power, but especially in the wake of WWI and the League of Nations mandates, this imperial authority was starting to waver.
Now, to say Ian Fleming was prejudiced would be putting it mildly. Racism, sexism, and homophobia pervade his work. From Live and Let Die in 1954, "'Shut yo mouf,' said the negro, but he pulled Bond's hand an inch or two down his back", to From Russia with Love in 1957, "All women want to be swept off their feet. In their dreams they long to be slung over a man's shoulder and taken into a cave and raped", to The Man with the Golden Gun in 1965 published shortly after Fleming's death, "Now it may only be a myth, and it is certainly not medical science, but there is a popular theory that a man who cannot whistle has homosexual tendencies". Indeed, trying to pin to Fleming's racism and prejudice down to just Korean people is impossible, because he has said equally horrible and prejudiced things about the Chinese and Black people like in Dr. No in 1958, "Not that they don't take the black girls when they want them. You can see the result of this in Kingston -- Chigroes -- Chinese negroes and negresses. The Chigroes are a tough, forgotten race", the Japanese like in Quantum of Solace in 1960, "A violent people without a violent language! I must write a learned paper on this. No wonder you have nothing left but to commit suicide when you fail an exam, or cut your girlfriend's head off when she annoys you.' Tiger laughed, 'We generally push them under trains and trams'", and women in, well, all of his work.
This extreme racism, sexism, and homophobia wasn't made in a vacuum nor was it merely a product of his upbringing. Pearson points out that Fleming was writing to a particular audience. His stories actively pandered to the patriotic common English person and the British elite who looked with great nostalgia at what the British Empire used to be. And indeed Fleming did this by kind of repackaging pulp stories from his childhood like Bulldog Drummond and the stories of Richard Hannay which both heavily feature British patriotism, espionage, a kind of heavy handed machismo, and of course, racism. Cook goes on to further state in his article that the new liberal elite of the era had begun to decry these old stories as relics of the bygone era. Fleming created a new Bulldog Drummond for the Jet Age, and it still features a very heavy prejudice towards really all non-WASPs as well as heavy sexism. This goes to show that Fleming was writing to not only a nationalistic audience, but one that was still willing to accept an extreme prejudice and racism in their fiction.
In short, Fleming was raised in a very prejudicial time, was incredibly racist and prejudiced in and of himself, and was able to express that racist and prejudicial sentiment in his work due to the receptive nature of the British reading public and it's nostalgia for a time when the British literally ruled over a bunch of people of different skin colors and ethnicities via an oppressive imperial bureaucracy. However, that prejudice was not specific towards Koreans nor exclusive to a specific time period shortly following the Korean War.