In the case of Belgium, it remained loyal to the Belgian government-in-exile in London.

France's possessions were more tricky. With France having essentially 2 competing governments (Vichy and the Free French), each one had to make a decision as to whom they were going to be loyal to. For instance, French Indochina was loyal to Vichy, and was occupied by the Japanese as a way into SE Asia. Madagascar was seen as a potential invasion target by the Japanese navy due to that colony's loyalty to Vichy, and was invaded by the allies to prevent that. Pro-Vichy Syria had real potential as an Axis stepping stone into invading the middle east, and was a base for German bombers aiding a pro-Axis Iraqi rebellion against the British.

The most loyal Free French "colonies" were in central Africa. It served as an important source of recruits to their cause, and was a launchpad for early attempts at taking more loyal Vichy colonies in west Africa, which didn't fare to well initially.

The other tiny possessions in the Caribbean and the Pacific were very pro-free French. Tiny Saint Pierre and Miquelon, off the coast of Canada was split between the general population being Free French with the government being pro Vichy. That government was overthrown in a coup ordered by Charles de Gaulle. French Guiana in S. America had a similar story.

I might have missed a few places, but the story pretty much repeats regardless.