The origins of the Soviet nationality policy were a bit of a mess. Marx and Engels didn't really discuss nationality policy in their works, except to note that it was a bourgeois means to divide the workers, and it was seen as a step on the path to socialism. The early European socialists had major debates over how to work around nationalism, with two major schools of thought prevailing: the idea of territorial autonomy (a region populated by an ethnic group would be represented by said group, effectively a nation state), which was favoured by Lenin; or the idea of cultural autonomy (everyone of one ethnic group would have the same representation, regardless of where they lived; I believe Belgium has something like this for the Flemings and Wallons), which was promoted by the Austro Marxists, especially the Jewish Bund. Within the confines of Russian socialism, territorial autonomy won out, and was promoted in an essay Lenin commissioned a young Joseph Jughashvili to write in 1912, "Marxism and the National Question" (this article was published under his more famous pseudonym, Stalin, which he began to use almost exclusively). In the essay Stalin stated that "A nation is a historically constituted, stable community of people, formed on the basis of a common language, territory, economic life, and psychological make-up manifested in a common culture." This would make up the basis of Soviet nationality policy.
As to how the various regions became union-level republics (the top level; de jure independent states), and who were autonomous republics (within a union republic), was more a matter of convenience and practicality than anything really specific. By 1991 there was 15 union republics, though at the creation of the Soviet Union there was only four (Russia, Belarus, Ukraine, and the Transcaucasus, which was made up of Armenia, Azerbaijan, and Georgia). Turkestan (which is the five Central Asian states of Kazakhstan, Kyrgyzstan, Tajikistan, Turkmenistan, and Uzbekistan) was nominally part of Russia as an autonomous region, while the three Baltic states (Estonia, Latvia, and Lithuania) were independent states until being annexed in 1940, and Moldova was still mostly part of Romania at this time (it would be incorporated during the Second World War).
For six of the fifteen, it was a simple manner that they had previously been independent: the three South Caucasus states and the three Baltic ones became independent in the aftermath of the Russian Revolution, and as they all effectively represented a distinct ethnic group, it was logical they would retain that status (the South Caucasus were unified between 1922 and 1936 as the Transcaucasian republic, but even then all three were autonomous republics within it, and this idea was scrapped). A similar situation happened for both Ukraine and Belarus, which were nominally independent in 1917 (albeit as German puppet states), while representing distinct ethnic groups.
As for Central Asia, the initial idea was to keep Turkestan as one entity, but this was discarded in 1924, as it was determined that the major groups in the region (Kazakhs, Turkmen, and Uzbeks) should have their own republics. This was did not go smoothly as Central Asia was very intermixed, and the people there did not often identity along ethnic lines. Furthermore, within the region that became Uzbekistan was a significant Persian minority, and while they were initially given an autonomous region this was upgraded into a full union republic in 1929 as Tajikistan. Similarly, the Kazakh republic was initially called the Kirgiz republic and was an autonomous republic within Russia, but in 1925 it was renamed the Kazakh republic and a new Kirgiz republic (what became Kyrgyzstan) was formed within it; this new Kirgiz republic was upgraded in 1936 to a union republic.
And lastly I'll touch on Moldova, which existed as an autonomous republic within Ukraine form 1924 to 1940. That year it was given the territories of Bessarabia and Bukovina that the USSR acquired from Romania, and was thus upgraded to a full union republic.
Now why didn't places like Tatarstan, or Bashkiria or wherever get to have full union status? One of the arguments put forth is that it did not serve the goals of Soviet nationality policy. Several scholars have argued that the states listed above all largely got their status because they all had international borders. This worked for propaganda purposes, as in many cases there was large numbers of the titular ethnic group in the neighbouring country (Poland had a lot of Belarusians and Ukrainians; Persia had Persians, which the Tajiks are, and Azerbaijanis; Afghanistan had Persians and Uzbeks; China had Kazakhs and Kyrgyz). This is what Terry Martin called the "Piedmont Principle" (based on the region in Italy that was a focal point of the unification of Italy in the 19th century). Thus these stats were given "independence" (technically speaking the Soviet constitutions did grant this), while regions within the Russian SFSR (and other autonomous republics, like Abkhazia, Nagorno-Karabakh, and Karakalpak) were not so lucky.
For reading on this:
The Affirmative Action Empire: Nations and Nationalism in the Soviet Union, 1923-1939 by Terry Martin (2001) is easily the best book on the subject.
Empire of Nations: Ethnographic Knowledge and the Making of the Soviet Union by Francine Hirsch 2005) is a good compliment, and she touches on the efforts to identify the peoples of Central Asia and then divide them (something Martin doesn't really do; he focuses on the west, mainly Ukraine).
The Sorcerer as Apprentice: Stalin as Commissar of Nationalities, 1917-1924 by Stephen Blank (1994) is also worth looking at, as he specifically looks at the cases of regions like Bashkiria and why they were not upgraded.
A State of Nations: Empire and Nation-Making in the Age of Lenin and Stalin edited by Ronald Grigor Suny and Terry Martin (2001). This is a collection of essays that touches on most of the above as well, and gives some really great context to how the Bolsheviks came up with their decisions.
I recently found answers by /u/Kochevnik81 and /u/Noble_Devil_Boruta for How dominant were Russians in the USSR? but hopefully someone else could write an original answer for you.