It depends on which part of the empire you are talking about. Its also worth noting that France and the government of France didn't technically "fall." The country we usually refer to as "Vichy France" was the legal continuation of France and the its leader Petain had been legally transferred power, granted he abused it to make himself more of a dictator, but he was still given his initial position legally. Anyways after France capitulated and signed the armistice at Compiegne, a call went out to all of the French colonial administrators to remain loyal to the government of France, and most did. Its interesting to note, that at the same time that they remained loyal, many French colonial governors wanted to continue the war against Germany from the colonies, most likely because they were unaware of just how severe the situation was on mainland France. DeGualle was relatively unknown when he made his famous "June Appeal" in London, and people like Churchill only put any faith in him because they hoped he might attract bigger, more influential French leaders to the cause. He failed to do so, he did attract the attention of a high profile French leader named General Georges Catroux, who was governor of Indochina, but he was sacked by the Vichy government before arriving in England. Two main reasons why most French leaders seemed to stay loyal to the Vichy government, was because there was a general sense of war weariness among the French who figured Germany would win a quick victory over Britain and they didn't want to stay in a losing war with Germany; secondly, the French opinion of Britain was soured with the attack on Mers-el-Kebir.

DeGualle sent appeals to French colonial governors asking for them to rally behind him, but he received a cold shoulder from most. The only bright spot was Central Africa, where the colony of Cameroon rallied behind the Free French forces. DeGualle received a tremendous reception in the capital of the Cameroon. DeGualle used the base of Cameroon to invade Vichy colonies in French central Africa, and he soon had control of Gabon and Chad. DeGualle next targeted West Africa, but a disastrous attack on Dakar was repelled by Vichy forces. West Africa would later join Free France in late 1942 when the tide turned against the Axis.

In the Asia and Pacific region, as well as North America the French had minor colonies that were generally out of the way and hard to communicate with. The colonies in the Pacific sided with Vichy initially with governor issuing a decree that all administrators must take an oath of loyalty to Petain , but eventually an informal referendum was held in which the population of the colonies clearly supported Free France; this eventually led to the governor being ousted and the colonies of the Pacific joining Free France. The islands were important bases for the Americans in their war against Japan. The French have a small island colony off the coast of Newfoundland named "St.Pierre and Miquelon." DeGualle "invaded" the islands (it was a small force of 3 ships, but the Vichy governor surrendered immediately) which caused a diplomatic incident with the USA, as the Americans were trying to stay neutral with the Vichy government and didn't like DeGualle invading islands so close to the USA.

Another major colonial area was Syria and Lebanon. They were home to a huge Vichy garrison, and DeGualle wanted to take them back. He pestered the British to let him invade, but they kept refusing. However things changed when the British puppet state of Iraq revolted, and the Germans used Vichy airbases in Syria to support the revolt from the air. The British and Free French invaded and Syria and Lebanon fell to Free France. A similar fate would befall the colony of Madagascar, where British forces invaded and installed a Free French governor.

Indochina remained in Vichy hands, but was taken over by the Japanese who left the administrators in place, this would continue until 1945 when the Japanese fearing the administrators had turned Free French, overthrew them and set up a bunch of pro-Japanese puppet states.

In the end, the ony major colonies that remained completely loyal to the Vichy government was the North African colonies of Algeria, Morocco, and Tunisia, and technically Indochina, but as already stated it was essentially taken over by the Japanese.

Just to add a source, Julian Jackson's "France: The Darkest Years"