Yes, Nazi Germany was a totalitarian dictatorship, so the government was highly centralized. The overarching guiding principle of the German state during the Nazi era was the "Führer principle" (Führerprinzip), which essentially meant that Hitler's word was the ultimate source of policy, rather than the written law. The Nazi bureaucracy and the relations between its leading figures was complicated to say the least, but all legal and political questions were ultimately to be answered based on the Führer principle.

In terms of the actual organization of the government, the Nazis never formally abolished the Weimar Constitution after they came to power; after the passage of the Enabling Act in March 1933, which allowed Hitler to govern by decree and gave him essentially unchecked power, there was really no need to abolish the Constitution since it had already been rendered irrelevant. The Weimar Constitution had established a federal structure for the German state, where power over various governmental responsibilities were divided between the federal (Reich) government and the states (Länder), as is typical of federal systems. The Reich government had control over foreign relations, immigration, trade, commerce, defense, and communications services, while the state governments were responsible for matters like education and other issues not covered by the Reich government; Article 12 provided that the states had control of all matters not directly controlled by the Reich government, similar to the Tenth Amendment to the US Constitution. However, as in the US federal system, there was a supremacy clause, which stated that Reich law would supersede state law in the event of a conflict.

The federal system established by the Weimar Constitution technically remained in existence during the Nazi period. The Nazi government continued to hold sham elections for both the federal parliament (Reichstag) and the state legislatures (Landtage), maintaining at least some veneer of a democratic process and popular support even though Germany was already a fully-fledged totalitarian dictatorship. There was at one time a plan to formally abolish the federal system, but it was never carried out (although the Prussian government was under direct Reich control, as it had been prior to the Nazi seizure of power). Instead, the powers of the states were gradually eroded and the actual state governments were largely superseded by the Reichsgaue, which were party districts that were overlaid on the existing federal structure. Some issues, like education, were left in the hands of the lower administrative levels, but these powers were gradually usurped by the Nazi Party, leaving the state governments with little real power. The same process was carried out in the private sphere, with the state and party taking over most private associations (sports clubs, social clubs, etc.). This consolidation of power over all aspects of German society by Hitler and the Nazi Party is known in German as Gleichschaltung, which is a difficult word to translate directly, but is generally understood to mean "coordination" or "consolidation". This process was carried out relatively quickly after the Nazi seizure of power, and it is generally understood to have been completed by the time the Nuremberg Laws were passed in September 1935.

I will point out that I don't think it's a good idea to apply the concept of "states' rights" in an American sense to the German context, because the origins of the German and American federal systems were quite different and most of the similarities are superficial. You can't really extrapolate from the German to the American context in a useful or historical way. As far as the Soviet Union goes, although both countries were totalitarian dictatorships dominated by a single ruling party, that's about where the similarities end, as the governmental structures and policy-making processes were very different, and comparisons between the two in terms of political organization are of minimal use, in my view.

Sources:

Text of the Weimar Constitution (English translation)

Map of German administrative divisions, 1944

Richard Evans, The Third Reich in Power (Penguin, 2005)

Ian Kershaw, Hitler 1889-1936: Hubris (Norton, 1999)