So to answer your questions (to the best of my ability) one by one...

How close in culture were the Angles and Saxons?

Probably pretty close. Their homelands were right next to each other, and their languages would have been very similar--both are typically grouped as part of the Ingvaeonic/North Sea branch of the Germanic languages. In fact, rather than saying there were "Anglian" and "Saxon" dialects of West Germanic, it probably would have been more of a dialect continuum. It should be noted that the Saxons seem to have been a far larger, more important tribe than the Angles, as many remained on the continent even after the migrations (regarding the Angles, Bede tells us that their homeland--roughly equivalent to the old Duchy of Schleswig--was completely deserted, implying that they left wholesale for Britain). The remaining Saxons eventually wound up causing trouble for Charlemagne. Lower Saxony in Germany still carries their name.

Why were their groups the most prevalent of the northern Germanic tribes who moved to England?

We're not really sure. I've heard theories that the fact that their land was rather poor, marshy, and prone to flooding might have played a role.

What happened to the Frisians and Jutes?

Who the Jutes actually were is up for some debate. They are generally considered to have originated on the Jutland peninsula, as their name would strongly imply, but there is disagreement over this. Bede tells us that they settled Kent and the Isle of Wight, which has sealed their name into history, but the material culture found by archeologists in Kent seems to be more consistent with that of the Franks than anything. The Frisians, meanwhile, are mentioned in an early document by Procopius, who states that they were one of the three great nations on the island of Brittia (Britain) along with the Britons and the Angles, but after that they largely disappear from the record. Why their identity was lost isn't really clear. Some place names have been cited as evidence of Frisian migration (Frieston in Lincolnshire, for one). For what it's worth there is also place name evidence of some other Germanic tribes making the crossing as well. Swaffham in Norfolk is widely agreed to mean "homestead of the Swabians," and I've read at least one paper that posits that Wendlebury in Oxfordshire derives from the name of the Vandals.

When and how did the Angles and the Saxons become the "Anglo-Saxons"?

The term "Anglo-Saxon" seems to have been used by writers in continental Europe to make sure the Saxons in Britain weren't confused with those who remained on the continent. Alfred the Great apparently sometimes referred to himself as the Rex Angul-Saxonum, which arguably represent the first time it was properly used as an ethnonym.