So, recently in history class, my teacher said that when a king dies and the person who takes over is not his son or daughter it is considered a dynasty change. He used the example of Frederick William IV of Prussia and his brother William I. Sorry if my english is bad as it is not my first language.
Not really sure how to source this other than as general knowledge, but William I and Frederick William IV were brothers and both were members of the House of Hohenzollern. I'm not familiar with any scholarly works that consider William's accession to the throne as a change in dynasty.
Generally speaking, in Europe, the change comes when a new family (or branch of the family) takes over. So, for example, in England/Great Britain, when the Tudors took over from the Lancasters, or the Stuarts from the Tudors. But even then, all of the families were interrelated.
Sometimes there isn't a clear rule to it either. For example, Queen Victoria was a member of the House of Hanover, but her son and heir, Edward VII was of the House of Saxe-Coburg-Gotha, through his father, Prince Albert (Victoria's husband). As it would happen, that family is still on the throne (albeit with a name change to Windsor, thank you World War I). Prince Phillip's (Queen Elizabeth II's husband) house is actually the House of Glücksburg, but far as I know, the future King Charles (or William, as it may be) will keep the House of Windsor. So there doesn't seem to be a good rule at all.
Now, this all changes when a family dies out (think the Spanish Hapsburgs) or is overthrown (like any number of dynasties, such as the Ming Dynasty in China). Then the change is easy enough to determine.
Again, not really sure how to source this. This isn't really the sort of thing that historians write books about, far as I know.