The conferences that took place during WW2 between the various powers decided that japan would be stripped of her overseas colonies, including agreement on korean independence, but there were no specifics decided on the implementation. In other words there was no decision on who would occupy korea (if at all) and certainly no decision of a split.

When the USSR declared war on japan they advanced faster than the american's seem to have expected. They feared that if the soviets weren't stopped they might face communist troops right across the water from japan. At the time Stalin was asking for an occupation zone in japan, same as in germany, but it's apparent that the US wanted to maintain sole control of japan. Therefore the scenario of a communist country just across the narrow korea strait was highly undesirable.

The US had no troops that could make it to korea and plant boots on the ground before the soviets reached the sea so they made an offer to split occupation zones along the 38th parallel in the hope that the soviets would accept. To their surprise the soviets accepted.

Overall the approaches that the US adopted towards korea post-WW2, i.e. split, occupation, and subsequent exclusion of ROK from Dean Acheson's defence line (that is said to have encouraged north korea to start the korean war), can be summed up as hasty instead of being planned.

From the perspective of how a soviet occupation of the entire peninsula would have affected the present day, I'd argue that it would've been better because the north and the south always had a co-dependent relationship. Resources and heavy industry in the north, agriculture and light industries in the south. Without each other they struggled, particularly south korea, but that's getting away from the topic.