It's specifically related to Central Asia as opposed to the Caucasus, but you might be interested in an answer I wrote about the process of Soviet territorial delineation for nationalities.

What's important to keep in mind is that the Soviets actually made a good faith effort to draw borders that matched ethnic populations, although this was never an exact science, and local communities and even local party structures argued endlessly over where borders should be exactly (often because of resource needs or wants).

With that said, these were borders of Soviet Socialist Republics within the USSR - they were never meant to be international borders between countries not united in a greater socialist project. In the case of Armenia and Azerbaijan in the 1920s when these borders were delineated, they weren't even Soviet Socialist Republics, technically, but autonomous republics in a Transcausian Socialist Federative Soviet Republic, which in turn was a member Republic of the USSR. They were part of a federation within a federation.

Not to break the 20 year rule, but part of the reason why the current border conflict is happening is specifically because the border was never demarcated to a precise degree that international borders usually are.

One last thing: I'd push back against population transfer being seen as an "easier" alternative. In the case of India and Pakistan the partition and population transfer caused the mass migration of millions and deaths of hundreds of thousands (at least). And it didn't prevent both countries from fighting four wars over border disputes. Similarly the 1988-1994 war and 2020 war in Nagorno-Karabakh saw the mass flight of Azerbaijanis and Armenians respectively.

Nagorno-Karabakh and Nakhichevan have very different reasons for existing, though both do complicate things in an already complicated region. And despite what some may think, it was not a deliberate attempt by Stalin (or anyone else) to "divide and conquer" the region; that stems from an outdated view of the formation of the Soviet Union (most prominently backed by Richard Pipes), and does not reflect the reality on the ground.

Nagorno Karabakh had long had an Armenian-majority population, though due to geography it was largely cut off from the rest of Armenia, and this did not change with the establishment of the Republic of Armenia in 1918, nor in the subsequent Bolshevik annexation in 1920. The region is quite mountainous (hence the name "Nagorno", which stems from the Russian word for highlands or mountainous), and it was only in the 1990s that a proper road was built between modern Armenia and Nagnorno Karabakh (financed by the Armenian Diaspora). In the 1920s this was even more difficult to work around, and thus most communication went east (towards Azerbaijan) rather than west (towards Armenia), which during the Russian Empire times made no difference.

When the Bolsheviks took over (they annexed Azerbaijan first, in April 1920; Armenia followed in December 1920, and finally Georgia in February 1921) they did initially look at incorporating Nagnorno Karabakh into the newly-formed Armenian SSR, despite the troubles it would face. There was quite a bit of discussion between the leading figures at Narkomnats (the Commissariat of Nationalities, led by Stalin), Kavburo (the Caucasian Bureau, which oversaw the annexations; led by allies of Stalin), and the Bolshevik government (led by Lenin). Things were also complicated because the Red Army did not have full control over the region, even after they annexed everything, as the previous independent factions still had strong support.

By May 1921 Bolshevik control was consolidated in the region, and a border commission got to work to finalize things internally. They first agreed to have Nagnorno Karabakh join Armenia, a decision strongly opposed by the Azerbaijani Bolshevik government (who did not want to lose territory, especially to Armenia). However on July 5, 1921 a new policy was put forth: keep Nagnorno Karabakh within Azerbaijan, but give it autonomous status. The reason for this about face is not clear, but Arsene Saparov suggests that with the final rebellious region of Armenia (Zangezur, near to Nagnorno Karabakh) finally under control, there was little need to placate the Armenians anymore. And as I noted, it was more practical, geographically speaking, to have it within Azerbaijan. That said, it was not until 1923 that the Nagorno Karabakh Autonomous Republic was actually created, so there was still some final details to work out; Saparov suggests this was because determining what and who exactly should be incorporated into the region was quite difficult: there was no borders delineating the region prior to this, and census data was not that accurate, as it dated from 1886.

As for Nakhichevan, this has more to do with the Soviet border with Turkey. Shortly before Mustapha Kemal declared the Turkish Republic, a treaty was signed between the two states, ensuring they would have peaceful relations (recall the Ottoman and Russian Empires were on opposite sides during the First World War, and indeed had been fighting each other for centuries). One condition was that Turkey have a land border with Azerbaijan, as they are both Turkic peoples and have a lot of cultural similarities. As it had a strong "Tatar" population (to use the phrase for largely Muslim peoples of the region; most would be considered Azerbaijani today), it was a logical move, and mollifying Turkey was seen as a good thing, considering the Bolsheviks were engaged in Europe still. The decision to incorporate Nakhichevan into Azerbaijan was also enshrined into the 1921 Treaty of Kars, with Article 5 stating:

The Turkish Government and the Soviet Governments of Armenia and Azerbaijan are agreed that the region of Nakhchivan [sic], within the limits specified by Annex III to the present Treaty, constitutes an autonomous territory under the protection of Azerbaijan.

Lastly I'll quickly touch on nationality as it pertains to the Bolsheviks, as that played a part here too. Marxism argues that nationality is a product of the bourgeois meant to distract the proletariat and keep them divided, an artificial means to split workers from each other. Ideally the people would move beyond such a concept and no longer identify themselves as an Armenian, an Azerbaijani, or whatever, but simply as a person. However after coming to power Lenin and co realized that this was not so easy to implement in the former Russian Empire, which had over 100 nationalities, many who were considered undeveloped. So they revised their plans and set about developing national identity, seeing this as a crucial step in their development towards socialism and the path to communism. So while national territorial units were established (like the Armenian and Azerbaijani SSRs), it was meant to be a temporary fix, and would one day be eliminated. Of course that never happened, and served the exact opposite role.

For some reading on the subject:

• Arsène Saparov, From Conflict to Autonomy in the Caucasus: The Soviet Union and the making of Abkhazia, South Ossetia and Nagorno Karabakh (2015). This is probably the most in-depth look at the formation of the Nagorno Karabakh ASSR, with two chapters on it specifically (covering 1918-1921 and 1921-1925, respectively).

• Firuz Kazemzadeh, The Struggle for Transcaucasia (1917–1921) (1951). This is an older book, but still the best one-volume look at the period between the Russian Revolution and the Bolshevik takeover of the South Caucasus. It gives some background on the relationship between Armenia and Azerbaijan, though Nagnorno Karabakh is not really touched on (it was not an important topic until the conflict began there in 1988).

• Thomas de Waal, Black Garden: Armenia and Azerbaijan through Peace and War (2003). The most comprehensive look at the Nagnorno Karabakh war up to the point of writing, by one of the leading scholars on the topic (who is critiqued both by Armenians and Azerbaijanis for his coverage for being biased towards the other). While it's mostly on the modern era, de Waal looks at some of the older background, though not as in depth as Saparov (who conversely doesn't look at much beyond that era).

Edit: I modified the wording to reflect that the region in discussion was specifically "Nagorno Karabakh"; earlier I had not made that as clear, and should have, as the Karabakh region is wider than the disputed region (of which the self-proclaimed government, and many Armenians, refer to as "Artsakh").

Since I've been rereading a lot on the 1918-1920 era in the last few weeks, just a couple of points to add to the more specific answer of /u/Kochevnik81.

Armenia was in a terrible strategic situation at the end of World War I. In fact, one of the truly strange actions Woodrow Wilson took after his illness in April 1918 - one that pretty much all observers agree marked a substantial change in his negotiating tactics - was to offer the United States as the mandatory power over Armenia, albeit with the strong encouragement of Lloyd George (who along with others in the British government realized he wanted nothing to do with the region by that point) and the caveat that it'd have to be approved by the Senate. While country after country had lined up to request a United States mandate for months - American administration was considered vastly better than the alternatives - Wilson had studiously avoided doing so outside of a handful of Pacific islands, knowing that there was little political support for it domestically and that getting a 2/3rds majority vote was extremely unlikely, let alone the cost both politically and economically in stationing American troops far from home.

The underlying problems of Armenia as well as the other Caucasus republics at that point is laid out very bluntly by MacMillan in Paris 1919, and it's worth quoting:

"Armenia, Daghestan, Georgia and Azerbaijan had all declared their independence in the spring of 1918. The new states, shaky, poor, struggling to cope with refugees, might have survived the brigands, the deserters from the Turkish armies, the White Russian forces, disease and hunger. They might have settled the differences that led them to war with each other. They might have held off General Denikin, the White Russian, because he had to deal with the Bolsheviks as well. What they could not withstand was the combination of a determined Russian assault from the north and a resurgent Turkey in the south."

The Treaty of Sèvres had been drawn up to force substantial land concessions by the Turks to provide for a much larger Armenian state among others; on the ground, this proved not just unrealistic but disastrous at it provided much of the popular support for Atatürk who swiftly made the borders drawn up thousands of miles away irrelevant given his military success. Nothing came of the American mandate (it was rejected in the Senate in May by a wider margin than the Treaty) and Britain's support was limited to a few poor quality surplus rifles in mid 1920. By September 1920, Atatürk begins conquering Armenia; by mid-November they sue for peace to hold the remnant of land they have left under their control. Some bitter irony here is that around that time, a map arrives from Wilson - he's tasked by the treaty with finalizing the borders - that on paper awards 42,000 square miles of Turkey to Armenia.

At that point, Armenia's Prime Minister concedes that there was "nothing remain(ing) for the Armenians to do but choose the lesser of two evils": become a part of the Soviet Union. The Treaty of Moscow a year later confirmed the return of two Turkish provinces to Turkey, and the Caucasus borders became an internal issue of the Soviet Union.