The book I'm getting this from I just picked up for a dollar: Medieval Households by David Herlihy. So far it looks fantastic but I want to double check what he's claiming at the start.
To give more detail, the claim is that their "familia" consisted of either all of the possessions of the household or all of the people in it under the head, depending on the context, whereas the head himself was the "paterfamilias". In some contexts "familia" exclusively referred to the slaves. Herlihy says that he cannot find an ancient word for blood family as we would think of it.
That's more or less correct, familia does not really mean "family," it means "household," as in all the property and dependents of the pater familias (or of a woman sui iuris, though legally a woman is the beginning and end of her familia, since her children cannot stay in hers and therefore her familia consists of herself and her slaves). That includes the slaves of the household and the children of the pater familias, except for sons who have been emancipated, at which point they become their own patres familiarum. It also includes the sons and grandsons of all sons who have not been emancipated, though it does not include the children of the daughters of the pater familias, as those children go into the familiae of the sons-in-law. So as far as blood relations go the familia is restricted only to direct descendants and the male line. The familia also, legally speaking, does not include the wife (if she is married sine manu) of the pater familias or any sons who have been emancipated, meaning that a Roman would not speak of them as being in the familia even though they are of course members of what we would call the "family." Now, familia can be used to mean what we mean more or less when we say "family," which is to say direct blood kin (though again, you wouldn't normally use it of an emancipated son), but it's just as often used to mean only the slaves of the household and most often it just means it's legal definition, the household of the pater familias. It's not an easy word to come up with a catch-all definition for in English, though. For example, since sons in potestate are not legal persons they have no legal right to property, and therefore all their property (which would be theirs as part of their peculium) is legally-speaking their father's. Since the familia includes possessions as well as persons, that property is part of the father's familia, although outside of a legal context familia more often refers to the actual people (or slaves, who are legally both people and things) rather than things. It's a word whose meaning is often best understood by context