To answer the second part of the question, the system followed the male line so that although Victoria was part of the Hanoverian dynasty, upon the death of William IV in 1837, his brother, Ernest Augustus I, became head of the House of Hanover (and also monarch of Hanover, as the succession there was patrilineal) while his niece became Queen of Great Britain.

When Victoria married Albert of Albert of Saxe-Coburg and Gotha, his family line became the dominant one even though he was 'only' Prince Consort and not King. The name was then changed to Windsor as a result of anti-German feeling in World War I.

As to why the family name didn't suddenly follow this model when Princess Elizabeth married Philip Mountbatten (of the deep breath House of Schleswig-Holstein-Sonderburg-Glücksburg and member of the Greek Royal Family), bringing the Windsor dynasty to a close... it nearly did.

In order to marry, Philip renounced his Danish and Greek royal titles, converted from Greek Orthodoxy to C of E and became a naturalised British subject, taking the surname Mountbatten (his legal name until then was simply Prince Philip of Greece and Denmark).

He had lived in Britain since the age of seven, more or less raised by Lord Louis Mountbatten - Viceroy/Governor of India - the Philip's uncle and closest paternal figure since the fall of the Greek monarchy following the disastrous Greco-Turkish War. His maternal grandparents were Mountbattens, so it wasn't took much of a stretch.

With the marriage, Louis advocated the change of the family line from the House of Windsor to the House of Mountbatten, but Winston Churchill - who disliked Mountbatten for his political meddling - strongly objected, recommending that the queen issue of a royal proclamation that asserted the line would remain Windsor.

Philip complained in his diaries, "I am nothing but a bloody amoeba. I am the only man in the country not allowed to give his name to his own children."

A 1960 Order in Council established the surname for members of the family without royal titles as Mountbatten-Windsor. There's no real consistency to this as various members of the family have oscillated between using Windsor and Mountbatten-Windsor and when William and Harry were in school and doing their military service they used Wales (as in "son of the Prince of") as their surname. Ultimately it's hard to make out a pattern because circumstances where they would use their surname are so rare.

Sources: The Hanoverians: The History of a Dynasty by Jeremy Black, In Churchill's Shadow: Confronting the Past in Modern Britain by David Cannadine, Queen and Consort: Elizabeth and Philip: 60 Years of Marriage by Lynne Bell, Arthur Bousfield and Garry Toffoli and a hit from http://www.royal.gov.uk/ThecurrentRoyalFamily/TheRoyalFamilyname/Overview.aspx