There was absolutely no system whatsoever. After a brief attempt in the Napoleonic period, standardized surnames were only imposed in the Italian peninsula after unification in 1861, and in some remote rural communities surnames were only imposed decades later by census officials (when they didn't report back exasperated, having just given up).
That being said, we can certainly extrapolate some trends. Of course, the aristocracy and high bourgeoisie have the strongest notions of dynasty and family, and the retention of a definite surname was an important part of this, not in the least to ensure property and other wealth could be clearly transmuted through the generations. Large extended families were also important support networks on which a dynasty's prosperity was built, and a surname was an important marker of dynastic identity. Although branches of the same family migrating to different cities or merely taking up different professions could adopt variations or additions to their surname (or occasionally even an altogether different surname) it's common to see fairly consistent use of surnames when there is a vested interest to do so. Even in the aforementioned rural communities where there were no fixed surnames until the second half of the 19th century, the earliest adopters were the more prosperous community members who were interested in ensuring their wealth was transmuted to the next generation.
But if only those who had interest in transmitting wealth and dynastic identity used surnames, how did other people manage? In most rural communities, that is to say in most of the country, simple patronymics sufficed: Andrea di Davide is Davide's son, while Giuseppe d'Andrea is Andrea's son, and so on (with an apostrophe replacing the "i" to avoid two successive vowel sounds). Large families could be given collective patronymics in addition to individual ones: Giuseppe d'Andrea could be a member of the Andreini, Andretti, or Andreotti family (and it would be interesting to be able to verify if the adoption of a plural form surname is in any way correlated to the potential exasperation by the census official, as "Andretti" could be the answer to an official cutting corners by asking "What is your family called?" while "Giuseppe d'Andrea" would be the answer to "What are you called?"). Vestiges of this old patronymic system were still in place as late as the 1960s, as official documents would specify a first name, last name, and patronymic. To cite an example the footballer Gianni Rivera's original registration forms for the Italian football federation, for example, would have the following fields to fill out: Nome Giovanni (his given name), Cognome Rivera (his family name), Di Teresio (his father's first name).
In larger towns, exclusive use of patronymics could get awkward. There could be two heads of families called Davide, and they might both name their son Andrea. This is where locative or vocational surnames might be adopted: Andrea di Davide the blacksmith becomes Andrea Ferri, while Andrea di Davide who lives by the well is Andrea Pozzo. But of course, these need not replace paternal names entirely, and they could even be integrated with them: if there were two blacksmiths named Andrea, they could be referred to as Andrea Ferri di Gennaro and Andrea Ferri di Davide. Infinite variations could also be used to distinguish the two, with names like Ferronetti, Ferretti, and Ferrari bestowed on people as they communicated different things, reflected different personal preferences, or just rolled off the tongue better. Enzo Ferrari's grandfather, for example, was a blacksmith when the census man came by: had he gone into a different trade, people might be admiring cars called Scarpa.
Locative names were bestowed in a similarly functional manner. Leonardo da Vinci was famously baptized as "Leonardi di Piero." When he moved to Florence there might be several people named Leonardo whose father was called Pietro, so he became Leonardo da Vinci, literally meaning, "Leonardo from Vinci." Locatives could also be used to distinguish different branches of the same family who established themselves in different towns (the film maker Lucchino Visconti di Modrone comes from one such family). There's sometimes a distinction between the more aristocratic "di" (but only in the case of the locative addition to an existing surname) and the more plebian "da" (an aristocrat, with his dynastic ties, is "of" a place while a commoner is "from" a place).
In the pre-census period, families might have in interest in preserving a surname for a few generations, but then allow a new name to circumvent it. Sometimes, in a multi-part surname people could come to prefer using one part over the other. This could be personal preference or convenience. The famous Medici dynasty of Florence might have become known as Da Cambio if the banker known as Vieri da Cambio hadn't hired a cousin from another branch (the Di Bicci) who played up his being a Medici (and played down his being of the Di Bicci branch). It would be this cousin, Giovanni di Bicci de Medici, whose later success would enshrine the use of the Medici name as an identifier for that family as they came to prominence.