I can neither confirm or deny however I can point out that the reason why underwear was white was washability. In the late 19th century we see "wash dresses" as a category of daily clothing - meaning dresses from washable material. The process of laundry was the most effective when the material could be boiled which cause colours to fade. That said white and crisp shirts were considered a mark of gentlemen in the later part of the 19th century. Laundry was hard and time consuming and wearing white clothing only to dirty it up after one use does sound as something that only the rich can afford.

There's a nice description of how laundry looked like in middle class household in the book "How to be Victorian" by Ruth Goodman. She also says cricket was the most popular sport in Britain by 1860s - in all classes, even in working class - and that white clothes became "de rigeur" by 1850s. In her acknowledgements she mentions plenty primary sources however the book is aimed at popular audience and doesn't contain citations.

I doubt the choice of white clothing for these sports was a conscious choice intended to exclude working and middle class and even more so I doubt there will be any direct mentions supporting this theory in contemporary primary sources. It may have been a side effect of elitist behaviour regarding those sports as a whole as discussed here https://www.jstor.org/stable/3789301 (pdf here https://www.berkeleycitycollege.edu/mswiencicki/files/2013/09/Consuming-Brotherhood-Web-Version.pdf) so maybe try to explore works referenced in the appropriate section of the article.