Like many things that happened with Korea around the end of the war, it was a case of a lack of preparedness and planning.

The first “lack” was a lack of strategic planning. It had been decided at the various meetings between Roosevelt, Stalin, and Churchill, that the japanese were to lose territories conquered by force. That included Korea but saying that “in due course” it would be granted “freedom and independence”. The idea of trusteeship for 20-30 years was raised by Roosevelt but nothing was actually agreed.

The second “lack” was that no decisions were made as to who would be the initial liberating force, who would occupy, for how long, etc. Soviet forces joined the pacific war at the behest of the US but they advanced far faster than the americans had anticipated. The expectation had been that they’d take a while to deal with the japanese Kwangtung army, which was supposed to be a crack group, but the soviets made short work of them and advanced at speed. Remember that no agreement or plan had been made as to who would occupy what. At the speed they were advancing it seemed to them that they might occupy the entirety of the korean peninsula before american forces were near enough to land and the american planners realized that that would put the soviets right next door to Japan.

At that point the americans were fully intent on turning japan into their sole occupation zone, i.e. no sharing with the soviets even though Stalin was strident in his demands for a piece of Japan. They had a plan to occupy Hokkaido, the northern-most island of Japan but they had to settle for southern Sakhalin and the Kurils (which they still hold today). This attitude of seeing South Korea as a buffer zone protecting Japan coloured US strategic decisions for the next few decades. For example, initially South Korea was only meant to be economically enhanced to the point of avoiding communist uprisings and as a market for japanese heavy industry, essentially reviving the colonial economic setup from pre-45. It just happened that South Korea did far better in the first 5-year plan than even the wildest dreamers had hoped so credit became more readily available for future 5-year plans after that.

The third “lack” was any knowledge or understanding about korea. US troops were nowhere near close enough to land in korea to block the soviet advance so in desperation two low-ranking american intelligence officers, Dean Rusk and Charles Bonesteel, were asked to come up with a proposal to put to the soviets that would stop them short of the southern korean coast. They had no idea or background about korea. To be fair the entire US intelligence apparatus had very little idea about korea either, but I will say that of the country summaries from the time that I’ve read they were surprisingly good and lacked a lot of the negative bias towards koreans that was the norm of contemporary academic works.

What they did was literally pull out a National Geographic map and pick a line in it, about half-way down the peninsula. That was the 38th parallel.

They had no idea of how that would cut off the agrarian south from the industrial north, basically creating two crippled economies – one that couldn’t feed itself (i.e. north korea) and one that didn’t have the industry for basic self-sufficiency (i.e. south korea). Or how that would cut off north korean industry from the major source of labour, and how the south would find itself with massive unemployment. Had the partition been temporary then it wouldn’t have been as much of an issue but here we are.

They sent it to the soviets fully expecting them to ignore it and carry on. But to everybody’s surprise the soviets agreed. I don’t believe it’s ever been conclusively established how far they had intended to advance, or even if they intended to advance into korea at all because historically the russians had only been interested in manchuria and possibly warm water ports in korea but primarily in keeping korea as an independent buffer zone. It’s likely that the soviets were playing it by the seat of their pants just as much as the americans. It’s believed that their acceptance came partly due to a similar secret proposal having been discussed between the former russian empire and the japanese before the Russo-Japanese war broke out in 1904 centered around the 38th parallel and they seem to have thought that the americans were doing the same.

So the partition happened. Americans landed in August ‘45, well after the soviets had occupied the northern part. The fourth lack is that the americans had no idea of what they were going to do during the occupation and sent the worst possible candidate, John R. Hodge, a capable combat leader but entirely unsuited to take over the tumultuous political situation in korea. He just happened to be the nearest officer of sufficient rank.

The soviets weren’t any better for the koreans to be fair. But the eventual outcome was that the divisioin was cemented into stone, the korean war happened, and the rest is history.

The best resource for in-depth coverage is “The Partition of Korea after World War II” by Jongsoo James Lee. It’s explicitly focused on that short period and the relevant history leading up to it.