Let's say I wanted to go to Australia or an African colony. Could I just up and go or would I need special documents?
For the dominions, Canada, South Africa, Ireland (Northern Ireland and the Republic of Ireland), Ceylon (Sri Lanka), New Zealand, Australia, India (inc. most of Pakistan, India, Bangladesh and parts of the surrounding region) all British Subjects were also British Citizens and had no distinctive national Citizenship until the 1940s. This meant that a New Zealander (or an Australian, or an Indian) and a Londonder held the same passport (if they were privileged enough to have/require a passport at all) and had the same right of movement within the Empire.
The Commonwealth today retains this notion of equality of movement in many respects. Commonwealth nations are NOT considered Foreign to one another, in principle diplomacy is arranged through High Commissions and High Commissioners rather than through Ambassadors and Embassies. There is supposed to be freedom of movement between Commonwealth States, but obviously there are some cases where geopolitical realities make this unworkable.
Largely because of this relationship with Commonwealth Members, which includes two Nations which were not part of the British Empire; Rwanda and Mozambique, today's British passport holders (along with Sweden and Finland) can enjoy VISA free travel to 173 nations, more than any other countries (US, Denmark, Germany and Luxembourg have 172)
Another nifty symbol of the relationship between Commonwealth countries is that High Commissioners get four horses to draw their carriage when they visit Buckingham Palace whereas lowly ambassadors only get two.
http://www.atlasofbritempire.com/Commonwealth.html
http://en.wikipedia.org/wiki/British_subject#Prior_to_1949
http://en.wikipedia.org/wiki/High_Commissioner_(Commonwealth)
NB - I could not find hard info regarding the non-Dominion areas of the Empire for this particular period but I didn't find anything to suggest British Citizens were restricted in travelling around the Empire as a whole.
Edited - some corrections made.