The tl:dr answer to this question is yes, colourful clothes were likely available to a fairly large portion of the population--but with caveats. I'm going to take a look at a couple of the sources available to us to answer the question in more depth--that is, extant textile finds, and pictorial evidence.
Textiles and Clothing c. 1150-c.1450 by Crowfoot, Prichard and Staniland (published by HMSO) has a catalogue of some of the Museum of London's wool textile finds, which reflect a broad spectrum of fineness of weave and patterning (including plaids and checks). Current* colours include a variety of browns, pinkish-mauve, reddish-purple, crimson, pinkish-purple, yellowish-brown, red, orange, yellow, purplish-black, and buff. ( I say "current," as in many cases some textiles that are now brown may have originally been another colour and may have faded over the centuries due to being buried.) Some of these colours were likely obtained from more costly dyestuffs, but one of the dyestuffs that is fairly easy to identify in these textiles is madder, which produces a variety of reds, pinks, and oranges depending on the strength of the dyebath and the addition of mordants. Madder was a very common dye plant, but not a massively expensive one. Other easily-obtained colours from less-costly dyes would include yellows (from plants such as weld, dyer's broom, and a wide variety of plants),blues (from woad; the active chemical is a relative of indigo), and, of course, a wide variety of browns. The caveat here is that many of the less costly dyestuffs were more fugitive (subject to fading over time); furthermore, the same dyebath can often produce several rounds of dyed cloth--the first being the most intense, with each subsequent round being more subdued. As you might expect, cheaper cloth was more likely to be produced using either cheaper dyestuffs or these so-called "exhaust baths."
It is absolutely true that brown, specifically the colour known as "russet" (which may also be a type of cloth) was associated with the very poor (as well as being a colour associated with widowhood and penance).
What other evidence do we have of coloured clothing worn by the lower classes? Illuminated manuscripts are one excellent source--the Luttrell Psalter famously contains depictions of not just upper class people, but of labourers, entertainers, and craftspeople, and they were a whole panoply of colours. If you're willing to move forward a couple of centuries into the 16th century, take a look at the works of Pieter Brueghel the Elder, whose realistic "genre paintings" often focus on peasant life. You'll see reds, blues (light and dark), yellows oranges, and greens, alongside browns and creams.
So if I were attempting to construct an "accurate" set of clothing for a medieval labourer of, say, the 12th-14th century, I'd probably be looking at a more coarse woolen with some kind of linen undergarment, fairly conservative in cut (tighter sleeves, fairly roomy in the body (but not wasteful of cloth), perhaps dyed yellow or a muted red or orange, blue, or (yes) brown, with not much in the way of decoration.