Often the individual traditions started for completely innocuous reasons, and sometimes as a self appointed mark of distinction, such as the 28th North Gloucester's wearing of a badge on the back of their caps, as they managed to resist a cavalry charge in Egypt whilst in line, an almost impossible level of coordination.
Why these traditions became so revered is a little easier to explain in general terms. For the enlisted men, once they signed up, they would remain in the same regiment for their entire careers, though officers could exchange in and out. Through much of the 18th and 19th century enlistment would be for a 22 year period, though there was a 7 year enlistment introduced for the the latter period of the Napoleonic wars. Combining this with the fact that on paper there were very few differences between line infantry regiments, all these little traditions such as mascots and small uniform changes became an important part of the esprit de corps of the regiments, in a way that might not really make sense to outsiders.
Today the expectation for most troops is that only a few will remain in the Army for their whole careers, and changing operational methods means that even these soldiers will never be deployed on overseas duties for years at a time, but the traditions of the 19th century remain.