This depends on what you consider a colony to be. Colonisers can be countries and nations like Russia and China who are still controlling their colonies through national borders. Although I am guessing you mean in the traditional British and French sense.
Colonies are a monetary blackhole. To develop properly you had to invest money in extracting natural resources, train the local population to firstly understand your language and secondly understand how the colonists go about doing things, and lastly be able to survive when the colony eventually goes away. Even investing in all these fields does not ensure that you'll get a return for what you put in.
Only a few colonies were money making, such as British India, where they were basically in the right place at the right time, the land itself was wealthy and that the people could be exploited to the best of their abilities. Hence why British India was sending back monetary wealth to the UK due to having both the local population to support itself and the local economy and having natural resources to exploit.
Other colonisers such as Netherlands only found semi-wealthy areas, such as Indonesia, but others were harder to justify holding on to though colonisers had to be pried off of certain regions. Frances colonies for example in Africa did not produce a massive amount of wealth though they had resources which could be exploited.
In summary, a coloniser had to have a functioning economy with monetary wealth to spare, an understanding that whatever you put in you won't necessarily get back, and most importantly having a good location to colonise. Many countries did not have these resources or did not get into the colonial game soon enough to justify spending the amount of money necessary.
Sources: