A lord of a sizeable proportion would search for documents inherent to his family's ancestors in different ways.
Perhaps the most common one was to send people with legal education to scour archives and catalogues in both notary books and likely tribunals or monasteries in a given area in which the family might have been active in. Oftern it meant trying to find legal documents produced for a litigation (which are a very large amount of surviving documents), for a donation of material entity or for the grant of privileges to and from a certain place or personality.
Often, it happened that the documents found do not clearly and exactly state an ancestry or belonging to a branch of the family. Frequently they simply cite a person that acted in some way in the given context that the document was produced for. Those that were tracing ancestries elaborated on these bits of evidence to recreate a plausible descendance.

Of course, most of times ancestry was claimed without many pieces of evidence to prove it clearly, just to assess prestige or connect with someone. An example can be the Uberti family of Florence in the 1200s, which stated to descend from Catilina, but instead had their family head in man named Uberto which is found in a legal document from 1070 ca.