I'll preface this by mentioning that I won't actually be talking about the Victorian period; that begins in the 1830s, by which time the system of military uniforms had already been long established.

Uniforms with distinct colors date back to about the end of the 17th century; before, soldiers were typically responsible for providing their own clothing. However, to ensure the soldiers' health in the field, and to prevent officers responsible for clothing their men from skimping, governments began publishing regulations for the clothing of the men during the mid 17th century. These early regulations did not specify any particular color or style; the clothes simply had to be good. The French minister of war sent a sample coat to provincial administrators to simply serve as a point of reference for their procurement of clothing.

Making uniforms visually distinctive was an evolutionary step towards the flamboyant styles of the 18th and 19th centuries; if a soldier was dressed differently from the local population, it would be much harder for him to desert. However, eventually fashion superseded function; during the late 17th century, you see things like the first fashion magazines appear. Since in this period, military units were not only political instruments, but the more-or-less private property of their commanders, business ventures designed to turn a profit, flashy uniforms became a way for aristocratic officers to display their wealth as conspicuously as possible. The cut of the soldiers' coats was uncomfortable, and unsuited to ensuring his health. The three cornered cocked hat, originally a simple, wide brimmed hat to keep out sun and rain, could no longer perform its function with the sides pinned up. The cuffs of the uniforms could be an annoyance when the soldier loaded his musket, but they followed civilian fashion.

There's a great essay in John Lynn's Battle: A History of Combat and Culture dedicated to war in the 18th century, with a section on the development of military uniforms.