I've just done a quick scan for this claim and multiple sources do show that they, indeed, borrowed a lot of money.

Long story short, WW2 German economy was actually in bad shape. While Allied and German propaganda claimed it was amazing, the truth was that it was inefficient, their designs were overengineered, and they could barely cover their needs. They practically had to go to war to secure resources, weapons and other assets from other countries simply because the country was about to go bankrupt. War made people stop asking questions about the economy. There was a good deal of Western involvement and technology transfer as well, either through sympathisers or "normal" business. It was a curious capitalist-nationalist mix that Hitler labelled "National Socialism", creating his own, vague definition of socialism, where socialism stood for unwavering support of the German society and state.

The trick, however, is the crafty trickery they used to pull the rearmament off. You see, the loans themselves were hardly enough, so they resorted to clever financial and organizational trickery to pull it off without actually backing things. Of course it wouldn't be the only case of a facade like that; many economic approaches before and after relied on gaming the models and measurements rather than getting things done. Germany just got the war rolling before the ball dropped.

If you look deeper into it, you'll find how heavily Germans used captured equipment - sometimes even on the front lines, which is pretty much nowadays considered a surefire sign of desperation, for reasons best explained by logistics. Using captured equipment makes logistics more problematic and, obviously, means that resupplying it is more difficult unless the factories are still running. Even then, this excessive diversification results in many issues of logistics and makes organizing anything harder.

It's also important that while the popular image of Nazi military is that of a mechanized force, the majority of their units was non-mechanized infantry and most of their transport was horse-drawn. Now, this is not to say they didn't have some revolutionary ideas or such, but the truth was that it was all a gamble from day one and, in retrospect, they were doomed to fail precisely because of the weaknesses in logistics and economy; they could only really win a war where they took the opponent out before it was a factor and absorbed their assets to preserve momentum. War-focused economy is not able to sustain itself, and the loudly promoted civilian projects never really panned out. As such, with such a weak consumer economy, there was very little to draw funding from in the first place.

I may be a little rambly here, but I'm aware of two books: The Wages of Destruction by Adam Tooze, covering the whole German war economy nicely, and Hitler's Beneficiaries by Götz Aly was also brought to my attention and seems to touch on your mention of outside loans.