Canada and the US were colonized by English speakers from Great Britain much earlier than Australia, New Zealand, and South Africa.

English settlement in North America started all the way back in 1583 with seasonal fishing camps based in what is now St. John's, Newfoundland, Canada.^1 The first permanent year-round English settlement in North America was 1607 Jamestown, Virginia, USA.^2 Since then, the English dialects of North America and those of Great Britain have had over 400 years to diverge.

Compare this to the first English settlements in Australia (1788), South Africa (1806), and New Zealand (1841).^3,4,5 On average, these dialects have only had approximately 200 years to diverge from those of Great Britain. Moreover, by this point the British and North American dialects had been more or less separate for hundreds of years.

So no, the Canadian dialect seeming less 'British' to your ears is not a matter of American influence. It is simply because the dialects of the US and Canada—there are many, see /u/AOEUD's comments in this thread—have been developing 'independently' from the UK for a lot longer than the dialects of Australia, New Zealand, and South Africa. Furthermore, there are many dialects of English throughout the UK and no single 'British accent'. British dialects have continued to change in the hundreds of years since the era of colonization. Keep in mind that the dialects of the various English-speaking communities throughout the British Isles Great Britain have been diverging since way back when they were speaking Old English.

Languages gradually change over time. When language communities are separated by large geographic barriers (e.g. the Atlantic Ocean), their dialects will continue to change but in different ways. Dialects that were separated a long time ago will sound very different, whereas dialects that were separated more recently will sound similar. So, dialects that have been separated for 400 years (US & Canadian vs. British) will most likely accumulate more differences than dialects separated by only 200 years (British vs. Australian, New Zealander, and South African). This is the source of the difference you're hearing.

Despite this, people from Australia, New Zealand, and South Africa do not have 'British accents'.

If anything, they have Australian accents, New Zealander accents, and South African accents. Today, we generally use 'British' only to refer to people from the British Isles Great Britain, which combined with Northern Ireland makes up the bulk of the UK. To our Canadian and American ears, there are indeed some similarities between the forms of English we're likely to hear spoken by people from the UK, Australia, New Zealand, and South Africa. However, we should realize that these similarities are not universal and we shouldn't label them all 'British'.

Additionally, it might be more productive to think of these language differences as dialects instead of accents. Many people do use these words interchangeably, but others only use 'accent' to refer to the influence of someone's native language on a non-native languages. For example, someone whose first language is Japanese may speak English (as a second language) with a Japanese accent. We usually consider the different forms of English to be forms of a single language, because we can all still understand each other: English dialects are mutually intelligible (at least for now). As such, when we are examining the linguistic differences between various English-speaking communities it's useful to view them as distinct 'dialects' as opposed to the 'accents' of non-native speakers. They are all native English speakers, after all.

For more information, check out some past threads that address similar questions: https://www.reddit.com/r/AskHistorians/comments/xpd6w/when_did_the_current_dialect_of_american_english https://www.reddit.com/r/AskHistorians/comments/vi59g/when_did_americans_lose_their_brittish_accent https://www.reddit.com/r/AskHistorians/comments/umgc0/how_long_did_it_take_for_the_american_accent_to/ https://www.reddit.com/r/AskHistorians/comments/rlv3j/would_americans_at_the_time_of_the_revolution/ https://www.reddit.com/r/AskHistorians/comments/qohpe/before_the_american_revolution_is_it_true_that/ https://www.reddit.com/r/AskHistorians/comments/umgc0/how_long_did_it_take_for_the_american_accent_to/ https://www.reddit.com/r/AskHistorians/comments/1cms6k/have_american_accents_changed_over_the_past_60/

Sources

1. "Gilbert (Gylberte, Jilbert), Sir Humphrey". (2005). Dictionary of Canadian Biography Online. University of Toronto. Retrieved September 10, 2011.

2. "History of Jamestown". (no date). Apva.org. Retrieved September 21, 2009.

3. Hill, David (2008). 1788; The Brutal Truth of the First Fleet. William Heinemann, Australia.

4. Moon, Paul (2010). New Zealand Birth Certificates – 50 of New Zealand's Founding Documents. AUT Media.

5. Smith, Simon (1998). British Imperialism 1750–1970. Cambridge University Press.