What you're hearing as "traces of the accent" is primarily from a few features, non-rhoticity (not pronouncing r after vowels) and the trap-bath split. Both of these are innovations which didn't exist in the British dialects that became most dialects of American English, though many American dialects were non-rhotic. All dialects of English, American, British, Australian, etc, have innovations relative to what was spoken in past centuries, and many of their differences (including those) were rooted in dialect differences prior to colonization.
One issue here is that you're seeing these dialects as monolithic. But in reality, what you're probably thinking of as a "British accent" is actually one particular British dialect called Received Pronunciation, which is spoken on TV and is the prestige dialect. But the English of Bristol shares a great deal with American English, to the point that you might not recognize it as British. And some American dialects, such as those in New England, share a lot with the dominant British dialects of today, and had later influence of British English.
this sub has an awesome FAQ, use it.
Your question supposes there is one British accent and it's remained constant.
it also supposes there's a single American accent.
to this day, we probably sound closer than you think. For example, there was a new girl in my class when I was studying Russian in Russia. We spoke in class for a couple if hours, and I assumed she was American because her Russian accent sounded much more like mine compared to the German students in class. It wasn't until the break that I found out she was English.