Let's use the Capets for my example, because they're one of the biggest dynasties in history and there's a lot of information to be found of its members.
According to Wikipedia the dynasty can branch of into their own houses. Those houses are part of the main dynasty, but can also branch of into their own branches. This could continue for a few more branches and now you would be so far away from the main dynasty that I would think that they aren't a part of it anymore.
Take Denis of Portugal for example, he's part of the Portuguese House of Burgundy, which is part of the House of Burgundy, which is then part of the Capetian Dynasty. At that point you'd think that you would be that far removed as to not refer to the Capetian Dynasty anymore, but instead start a new dynasty for themselves.
Would Denis really been seen as a Capet? I know that the French Kings were, but I have a hard time believing that the same is true outside of France (or just any other dynasty in general).
What makes it a House, part of a dynasty, and not a dynasty on their own? E.g. Capet with Robertians.
As a side question, what kind of documents did European nobles and rulers in the Middle Ages use to verify their royal/dynastic lineages?