Given the lack of good sources corroborating the story that a bunch of little boys from Germany and France set out for the Holy Land after a boy in Cologne and another in Cloyes had a vision commanding them to travel to the Holy Land and peacefully convert the Muslims, and wound up being caught and sold into slavery in North Africa, as the traditional narrative goes. It seems more likely to me that instead of children, the puer in peregrinatio puerorum refers to irregular bands of servants/peasants/poor people as opposed to regular armies and knights. Am I totally off base here?
No, you're right but I'm not sure how many historians would say the Children's Crusade was actual children.
In the popular imagination there's a direct link between 'the Children's Crusade was Children' and George Zabriskie Gray's The Children's Crusade, published in 1871 and then repopularized in the late 60s (and republished in 1972). In hindsight, its popularization produces for the reader only secondhand embarrassment. At the time the then-popular anti-war left and at least some academics were motivated to make broad connections between the event and contemporary events. So a veritable cottage industry sprang up that put politics first and history second.
We get echoes of that uncritical acceptance of a very critically deficient work in things like Vonnegut's Slaughterhouse Five, whose original full title was Slaughterhouse Five or The Children's Crusade, and several other movies of similar branding.
But in terms of actual historians, few ever believed it was mostly or exclusively children. Over fifty Latin texts survive from the era, but very few of these texts are historically reliable and it is my understanding that taken together the event becomes more mythological than historical. In other words, historians know there was a popular "movement" that began in France, continued in Germany, and some form might've been sold into slavery but beyond the broadest outlines no one can say for certain because the sources are multitudonous and contradictory.
[...]
While nineteenth-century historiography accepted and sentimentalized the idea of the crusaders as children, scholarship from the middle of the twentieth century argued instead that the term pueri, the most common term used to describe the participants, should be understood as "peasants" rather than "children."
As you allude to, some historians have made the argument that puer originally was understood as poor of all sorts and that their child leader or leaders, if they really did exist, weren't really leaders at all but like many 'charismatic' movements they put the children forward as icons or mascots.
But, again, it's truly impossible to know
Did the Children's Crusade of 1212 Really Consist of Children? Problems of Writing Childhood History Scheck, Raffael. The Journal of Psychohistory; New York Vol. 16, Iss. 2, (Fall 1988)
Raedts, Peter (1977). "The Children's Crusade of 1213". Journal of Medieval History. 3 (4): 279–323.
Dickson, Gary (2008). Children's Crusade: Medieval History, Modern Mythistory.