The Empire was not in the public discourse to the extent you'd expect as it was mostly run by independent privateers and businessmen pursuing their own motives. Therefore, the empire was not a collective 'entity', but was instead a large collection of independents all under the British monarch. As a result of this decentralisation, it was merely referred to patriotically, and people didn't know the specifics of the empire. Even with this, the patriotic referral was a late nineteenth-century development.