Let's look at Caesar's account in its immediate litterary context (De Bello Gallico, VI, 14)

Of all these [insular] nations, the most civilized [humanissimi*, the closer to human nature*] are those who live in Cantium which is a coastal region in its entierty and whose custom do not differ much from Gauls. Most of those in the hinterland do not sow corn, but live on milk and meat, and are dressed in skins/furs. All the Britons, indeed, dye themselves with a plant which occasions a sky-like color, and thereby have a more terrible appearance in fight. They wear their hair longs, and shave every part of their body safe the head and the upper lip. Ten or even twelves have wives in common, especially between brothers, fathers and son. Furthermore when these have children, they're considering being those of the first that espoused her when still virgin.

Caesar is describing there a pretty much primitive folk, in stark contrast to the comparatively sophisticated Gaulish folk and their agricultural production and the people living in southernmost Britian (that he stated were not only closer to Gaul, but related to them genealogically and politically).
While ancient Britain wasn't exactly an agricultural powerhouse, we know trough other testimonies (such as Strabo or what was surviving from Pytheas') and especially archeology that insular peoples did knew farming since millenias (altough more continuously from the Bronze Age) or born tissues and sawn clothes. As Britain was particularily unfamiliar to Romans, against in contrast with Gaul, Caesar got away with fanciful descriptions of peoples that he didn't really dealt with (remaining in the southernmost regions) because that fitted the 'geographic' explanans on Barbarians established since centuries : further from civilization a population dwells, less develloped or partaking in 'normative' human activities it is, northernmost Britons eventually being described as living naked and literraly in swamps; peoples that, as for instance Germans, weren't really worth the trouble conquering.

Polyandry, as being utterly foreign and 'other', to traditional Roman conception of family, rather wells fit the general picture there especially in likening Caesar's account to Dio Cassius as he narrates a speech supposedly given by Boudica (Dio Cassius, Historia, LVII, 6)

For I rule over no burden-bearing Egyptians as did Nitocris, nor over trafficking Assyrians as was Semiramis (for we have by now gained thus much learning from the Romans), much less over the Romans themselves as did Messalina once and afterwards Agrippina and now Nero (who, though in name a man, is in fact a woman, as is proved by his singing,

Those over whom I rule are Britons, men that know not how to till the soil or ply a trade, but are thoroughly versed in the art of war and hold all things in common, even children and wives, so that the latter possess the same valour as men

In the same style of other discourses put in the mouths of defeated Barbarians by Roman historians , this draw a line between 'expectable' Barbarity from peripheral peoples that don't know better in their 'natural state' (especially as it's coming from their geographical furtherness) and Romans who degrade themselves morally and even physically in a way that Barbarians instinctively reject (in a reminiscent way to Tacitus' Germania considerations on corpores infames).

Does that means Caesar's accounts is entierely devoid of any truth?
It's worth pointing out that, while still firmly patriarchal, that women in ancient Celtic societies seems to have benefitted from a better social status than their ancient Greek and Roman sisters : Gaulish marriage is said to be contractual and with some preservation of the wives' interests (besides some possible matrilinear succession); Veleda, the seeress of the Batavian revolt, bears a Celtic name; Irish litterature accounts for 'contractual' intercourse (most famously with Mebd); ancient British women seems by the time of the conquest to hold some political weight by the description of Boudica and Carimandua's power, etc. and overall, there's possible instance of feminine agency in political or ceremonial matters.
Maybe this might have fueled Caesar's description of these 'further' Britons, more Barbarian than peoples he conquered. It remains largely speculative, however.

Eventually, it's pretty much likely Caesar's more or less made up British polyandry for the sake of both justifying his limited involvement in Britain and sacrificing to the ethnographic clichés about people too far from true civilization and normative humanity.