By "lord" do you mean the holder of one or more manors rather than one of the great magnates? If it's the former, Bruce Campbell (The agrarian problem in the early fourteenth century, Past & Present 188, Aug 2005) estimates the rural population c.1300 at around four million, 20,000 of them lords, or roughly 200 rural persons per lord.

HC Derby's estimates (Domesday England, CUP 1977, p 89) indicate a similar average in 1086: 7,100 tenants-in-chief and under-tenants and 265,000 recorded peasant households equating to upwards of 1.2m people, which Sally Harvey considers likely to undercount the agricultural total (Domesday England, in Agrarian history of England and Wales v 2, 1042-1350, CUP 1988).

An average of 200 may be on the high side if significant areas remained nonmanorialised, and it's important to remember that rural doesn't necessarily mean agricultural, so some portion would have earned their living from other occupations (Robert Allen suggests a fifth in 1300: Economic structure and agricultural productivity in Europe, 1300-1800, European Review of Economic History 4:1, 2000).

The average manor was of course rather smaller than this: for 1279 EA Kosminsky (Studies in the agrarian history of England in the thirteenth century, Blackwell, Oxford 1956) indicates an average of 13 peasant households in a thousand manors in a selection of southern, midland and eastern counties, which at five per household would suggest 60-70 individuals not including the demesne household and retainers.

For the greatest magnates numbers are of cours very different: Robert of Mortain's 797 Domesday manors (Chris Given-Wilson, The English nobility in the late Middle Ages: the fourteenth-century political community, Routledge, London 1987, p 8) presumably contained tens of thousands of villagers between them.