In a word, no. Despite being poorly recorded compared to most periods, we still have many contemporary written sources, as well as archaeological evidence. Books 5-10 of Gregory of Tours' Historia Francorum, for instance, concern events that happened within Gregory's own lifetime. Bede's Historia Ecclesiastica does for seventh-century Britain what the Historia Francorum does for sixth-century Gaul. Unfortunately, not all the sources from this period are so detailed, but they can still tell us a great deal. Gildas is our contemporary source for sixth-century Britain and, though his work has a clear political purpose, a critical reading can still tell us a lot about the political world he lived in. As you say, later sources such as the Historia Brittonum contain a lot of legendary material and can be treacherous ground in terms of interpretation (see David Dumville's works on this particular source). Others, such as the Anglo-Saxon Chronicle may come across as dry histories, but on closer inspection are highly untrustworthy for this period (the Anglo-Saxons had a habit of inventing historical characters and events to explain place-names). Not all sources are written on paper, however; Dark Age Britain has produced many stone inscriptions. One from Carmarthenshire commemorates a 'Vortipor, Protector of the Demetae'. This same figure is mentioned, and condemned, by Gildas, so we can see how Vortipor was viewed by his enemies and how he presented himself. Unfortunately, it is indeed rare for anything from this period to be doubly-attested.

Because of this, most of our reliable evidence concerning the Dark Ages comes form archaeology. It's difficult to overemphasise just how much can be learned from archaeology. It ranges from specific, ornate objects such as bracteates to architectural techniques. Pollen analysis even enables us to work out which land was being cultivated during the period. A single object, such as the Sutton Hoo helmet, can be so informative that it merits a whole chapter.

Sometimes, archaeology demonstrates some basis to legends. Tintagel and Dinas Emrys, for instance, were real Dark Age fortresses. At other times, it can prove that previously trusted sources are unreflective. Gildas's visceral picture of Saxons destroying British cities is not born out in the archaeology. Bede's claim that Kent was a Jutish land turns out to be something of an oversimplification, since the finds from west Kent are more Saxon than Jutish.

It's true, a lot of archaeological interpretation is just speculation, but it's often very carefully thought-out speculation, making use of the limited evidence available as well as historical parallels. It is unusual as a period of history in that most of the focus is on what happened, not why it happened. It requires both imagination but also a great deal of caution. What might have seemed like a perfectly reasonable assumption to historians and archaeologists fifty years ago might now be seen completely wrong. We make progress not only by digging up so much more new evidence, but also by examining our own biases, and those of our forerunners.