The flags represented the seniority of the admiral who's flag was being flown. The red ensign was the senior squadron. Initially Blue was the second most senior but after about 1650 White was promoted over Blue.

How this would work was a man would be promoted to rear-admiral of the blue. His next two "promotions" would be in rank but he would change squadrons. Then he would cycle through the squadrons as vice-admiral, and again as admiral, although for the longest time admiral of the red was reserved for the fleet admiral, and was held by only one man at a time. This changed in 1805 after Trafalgar as it became an award for exceptional accomplishment. So the promotion cycle would look like this:

RA-Blue -> RA-White -> RA-Red -> VA-Blue -> VA-White -> VA-Red -> A-Blue -> A-White (and after 1805 -> A-Red).

Traditionally any grouping of ships would fly the ensigns of their direct commanding admiral. But there were exceptions. Admiral Nelson for example had the entire fleet fly the white ensign at Trafalgar because he was in overall command. But otherwise there could be a mix of Red, White, and Blue ensigns depending on what admirals were present.

At the time admiralships were awarded based entirely on seniority. So an incompetent who managed to be promoted to captain would automatically be promoted to rear-admiral when a slot opened up. To prevent such men from actually commanding the fleet, they were instead promoted to what was known as the Yellow Squadron and were denied active commands in the other three squadrons. This functioned as a sort of forced retirement. So there was never an actual yellow ensign for them to fly.

He whole system was changed in 1864. The Royal Navy adopted the White ensign. The Red ensign was assigned to the Merchant Marine. The Blue ensign was assigned to reserve and auxiliary ships. And the distinction of three grades in each rank of admiral was eliminated.