As you note the primary blame can be assigned to the fact that Germany did not unify until much later than other European states. Germany had been the center of much of the controversy and wars in Europe's history. The Napoleonic Wars had been fought over German territory and the concern among many powers was that Germany would be too weak to hold back later French invasions. You can imagine a country that was considered so loose-weaved would not have the naval or administrative muster to manage vast overseas colonies.

Later after unification, Chancellor Bismarck would refrain from amassing overseas colonies recognizing that such efforts could make other European powers feel threatened. If they felt threatened, then those states might revert to warfare on the European continent against a Prussian state that had subsumed the other German kingdoms to become more powerful than any other single country in Europe. Bismarck calculated this not to be worth the risk and instead tried to create a more stable peace throughout Europe.

Source: Diplomacy by Henry Kissinger