The Bermudian accent is actually distinctive, and if you listen carefully to native speakers - especially in an informal context - you will notice it has certain Caribbean traits. Words like 'ask' and 'the' it's particularly prominent. Whilst you're right it certainly can sound a lot like an American accent, Bermuda is rich with its own vernacular too which helps to make it more distinctive. It's a sadly under-studied dialect to date but the consensus from linguists and historians is that it shows signs of very obvious Portugese, English, American and Caribbean influences; the American qualities you pick up may be something to do with how much American culture permeates through the island today, as well as their familiarity from your own exposure to American culture.

It should also be remembered that Bermuda's first settlers were the same English people who settled the future United States and brought with them their own accents that evolved into a similar but distinctive way of speaking; as a small island with a much larger white population than most of the Caribbean, the development of the Bermudian accent will have logically been more consistent, more isolated and probably took much more inspiration from the colonial white elite rather than from Creole vernacular. Being geographically far removed from the rest of the Caribbean (and Bermuda often considers itself to be wholly separate to the Caribbean region), there has also been less room for free cultural and linguistic exchange between the islands,

I think a good example of the accent's diversity and quirks would be Jennifer Smith, the Premier of Bermuda from 1998 - 2003. This is a speech given by Smith on the eve of the 2012 election to introduce the then-incumbent Premier, Paula Cox, as she was about to call the election. If you listen carefully, you can hear at times that the accent does have a certain distinctive Caribbean tone to it; at other times, it errs more towards British English or American. Likewise, Paula Cox's speech later in the video has similar features.

Incidentally, the accents of the British Caribbean are quite different too. Take for instance this video from 11:31 to 12:23 of procedings in the Parliament of Trinidad and Tobago; Kamla Persad-Bissessar (the woman speaking first), Esmond Forde (the acting Speaker) and Herbert Volney (the man who speaks last) all sound quite distinctive due to Trinidad's rich linguistic and racial history. Likewise, if you listen carefully, Portia Simpson-Miller's Jamaican accent in her 2011 election victory speech and Freundel Stuart's Bajan (Barbados) accent in this speech to the Global Forum, they're actually quite different. Definitely similar, but distinctive.

(Note: I've selected speeches by political leaders because they're public figures people from outside of their countries are most likely to know about, not to denote an endorsement of any particular political agenda or platform. And maybe someone will be introduced to the wonderful and fascinating world of Caribbean politics in the process, eh?)

The west of Ireland is also very prominent, in fact when you listen closely you can hear it quite a lot in these accents as Irish slaves and then later crews of the slave ships were mostly Irish, Google the Cork or Kerry accent and also Montserrat.