That really isn't how feudalism worked, assuming that the "feudal system" as envisaged by popular history ever existed at all. I'd recommend reading this excellent summary of current historiography around the use of "feudalism". Tenant relationships with the peasant classes did not constitute the ownership of people as property; they were a mutable series of negotiated obligations and responsibilities by both parties, predicated on acts of vassalage or homage. Freeman peasantry constituted a small, but significant proportion of the peasantry. Vassalage was also frequently commuted by moving to a city.