It's more right than you might suspect. Norwegian stave churches look quite different from continental churches. Their architecture was already established at the end of the Viking age. Basically the stave church is an architectural mix between a continental church (with its origins in both the Roman basilica and the Germanic 'hall') and something else. That something else might very well be a native Scandinavian style of temple-building.

The earliest Scandinavian churches (excluding the really early ones in Ribe, Hedeby, and possibly Birka) were often built at the same place as pagan temples. This is a likely reason why we still have not found the actual temple at Uppsala yet; it is below the modern church. Similarly, the church at the Danish 'copy' of Uppsala, at Jelling, is preceded by an older wooden structure. There is little reason to assume why the entire architecture would have changed too. Actually, many early churches (including the still existing Norwegian churches) have traditional art on it, such as dragons and whirly branches (Ringerike-style in the oldest Danish examples, Urnes-style on the later Norwegian ones). Wood preserved from an early wooden church at Hørning, in Denmark, also shows dragon carvings and preserved paint in black, yellow, and red.

Now, actual archaeological excavations of definite pagan temples are rare, but we do have them. There is one from Mære in Norway, or next to the manor in Tissø (Denmark) and Uppåkra (Sweden). These structures are only visible by the marks of their foundation posts in the ground, so they tell us little about how it would have looked above-ground. It is possible that later descriptions of these temples as 'girdled with gold' might be exaggerated, but it is quite likely that they would have been extensively decorated (just like almost anything Viking, Vikings loved decorating stuff). What we do know with certainty is that these temples are always square. This can be contrasted with normal house architecture, which is rectangular. There are always four massive posts.

Inside these temples, we sometimes find the stuff that was sacrificed there. In the 8th century these are often little gold foil images of a person, possibly the divinity to which they are sacrificed. Normally speaking, sacrifice would have been of animals or foodstuffs, though.

Now, these gold foil figures we also find in some big longhouses. They are often in the central 'hall' part of the house. It is quite likely that these hall spaces are the normal ritual spaces ('temples') of Vikings, and that these are the places where most sacrifices and rituals to the gods would normally take place. It now also makes sense why the temples are square: they are simply these special-purpose rooms inside the house, placed inside a separate building outside.

However, you should also be aware that not all rituals to the gods took place in temple-buildings. Placenames indicate that even by the late Viking age, it was groves, forests, springs, bogs, hills, ruins, and other natural places that were the primary focus for rituals. Possibly there would have been a single wooden statue of a god just somewhere out in the wild that would have functioned as a sacred space.