So basically how did the Soviet Union and Nazi Germany build all the infrastructure with literally no finances to do it with? Was it just all slave labour?
This is actually a really good question and one that I feel touches on a central aspect of what we now call Stalinism.
But to back up slightly and provide a bit of context: there was absolute uniformity amongst the 1920s Bolshevik leadership as to the desirability of industrialisation. This was rooted in an ideological predisposition, a fear of foreign invasion and an acute awareness of Russia's 'backwardness'. As Stalin famously put it in 1929:
We are becoming a country of metal, a country of automobiles, a country of tractors. And when we have put the USSR on an automobile, and the muzhik on a tractor, let the worthy capitalists, who boast so much of their "civilisation", try to overtake us! We shall yet see which countries may then be "classified" as backward and which as advanced.
But the dilemma remained: how could a backwards country afford to industrialise? Where was the investment to come from? It certainly wouldn't be capitalists driving industrialisation and, following the defeat of Bukharin and the Right, there was judged no time to allow for a strengthening of the domestic market 'at a snail's pace'.
Initially the problem was handwaved away in the first mad dash towards a socialist future. Financial controls effectively went out the window in 1928-31 as investment (largely based on printed money, see below) was poured into industry. This wasn't particularly sustainable but it wasn't until late 1931 that economic accounting (khozraschet) was re-introduced to Soviet enterprises.
With that initial period out of the way, the quasi-sustainable Stalinist economy that emerged from the first burst of industrialisation was largely financed via a combination of the below means.
Tax was the mainstay of the state's revenues, albeit with little direct individual taxation. Instead the key policy was the 'turnover tax', effectively an excise duty on consumer goods, particularly vodka. Other 'taxes' included a production tax on the profits of state enterprises.
Very substantial sums were also raised via the sale of state bonds to citizens. While not exactly a tax, these 'voluntary' bond drives were anything but that: real pressure was exerted at factory floor level to ensure that everyone bought in and contributed to the industrialisation campaigns. To put this in perspective: state bonds were the second largest revenue item in 1932 (after the turnover tax) and for the individual worker could amount to a month of their annual wages.
As mentioned above, liberal amounts of money were printed. Some effort was made after 1931 to maintain a stable currency but money in circulation increased by over 300% between 1928 and 1934.
Reduced costs. Key to Stalinist planning was the idea that economies of scale would reduce costs in industry, thus increasing the state's margin. This generally didn't happen but there was a concerted programme to instil a 'regime of economy', making workers work longer and harder for less pay. Quality also collapsed, which entailed hidden costs of its own.
A more prominent 'saving' however was Stalin's argument that a peasant 'tribute' (dan') would fund industrialisation. This is the assumption that the forced purchase of peasant grain, at an artificially low price, would enable the state to both feed the cities cheaply and export. This was a, if not the, major driver behind the collectivisation fiasco.
Foreign credit didn't play a major role in financing industrialisation but exports were key to earning the precious foreign currency that the Soviet state needed to balance its payment accounts. This in turn allowed for the purchase of foreign technology, materials and assistance. This was a vital concern but, unfortunately for the USSR, the Great Depression seriously restricted demand for Soviet goods/materials in the Western markets.
[Edit. Dammit. I somehow forget forced labour. Well I won't dwell on the point, except to note that the use of coercion and industrialisation were intertwined from the start. By 1940 some areas, eg the mining of rare minerals, relied almost exclusively on slave labour, while this was also a key part of the construction workforce. But this was one of the most inefficient components of a grossly inefficient system.]
That's a very quick outline of the main sources of finance for the Soviet Union during the 1930s. The key point I feel is that the vast majority of this capital was raised internally, by 'squeezing' the workers and peasantry so as to divert funds to industry. Many of the features that we associate with Stalinism - eg harshness, collapse in living standards, famine and shortages, etc - stem, I feel, largely from this relentless 'hyper-industrialisation' that demanded everything in the name of industry.
I'm not going to start talking about the Nazi experience, largely because it's not my field. There are definitely some similarities with the above however and I'd strongly recommend Adam Tooze's Wages of Destruction if you're interested in that experience.
Sources
Usually I recommend Alec Nove's Economic History of the USSR as the best introduction to, well, the economic history of the USSR. He does spend some time on the financing of the plans (that's where the printed money figures came from) but not a huge amount, if that's really what you're interested in.
For that we need more detailed works. And that's where RW Davies come in. His Industrialisation of Soviet Russia series is without par when it comes to the Stalinist economy; above I've drawn largely on Vol 3 (The Soviet Economy in Turmoil) and Vol 4 (Crisis and Progress in the Soviet Economy).
For the human cost of all this, particularly on the urban working class, I'd recommend Kevin Murphy's Revolution and Counterrevolution and Jeffrey Rossman's Worker Resistance Under Stalin for more qualitative accounts.
Germany had instituted a very heavy bureaucracy over their business sector as part of the very strict import regulations they enacted to deal with their critically short foreign cash reserves, which limited how much of the crucial goods they could import to keep their economy running. They used and expanded this to drastically shift production from consumer goods to military and heavy industry (which was then able to fund its own growth with profit and financing).
The government also borrowed a lot of money, first from foreign creditors such as the United States and then when they suspended payments on those from the German people.
While I can't provide an exact answer for the Soviet Union which I believe was mostly prisoner powered projects. I know for a fact that the Nazi government mainly used loans and citizens that were unemployed as a means to produce its major infrastructure such as the Autobahn.
Many that had been unemployed since the Great Depression as well as unskilled workers got forced into these work camps and were paid tiny wages with portions of that taken away to cover room and board. These construction projects were advertised as such that it would pay people especially the unemployed or those arrested for certain reasons e.g. rival political stances.