Without being rude, this is an incredibly modernist question. After a century of industrialised farming, it's very easy for us to massively underestimate just how labour intensive agriculture was for most of human history. The International Research Journal of Engineering and Technology ran a 2020 study on three modern tractors and plough assemblies (Rahate, Bokade and Wagh, Comparative Field Peformance of Tractors with M.B. Plough Attachment, IRJET 7.5, 2020) which found that, roughly speaking, all three of the tractors it tested (FWiW, the New Holland 3630, John Deere 5310 and Same Deutz Fahr Agrolux 55) could plow an acre of land in roughly an hour to an hour and a half. For reference, an acre is broadly recognised as the amount of land that a pre-industrialised farmer with a horse or oxen drawn plough and team of assistant boys could plough in a full day of work from dawn til dusk. As the character of The Ploughman in Ælfric's Colloquy states:

I go out at the crack of dawn to drive the oxen to the field and yoke them to the plough. For not even in the bitter winter would I dare to stay at home for fear of my lord; but, when I have yoked up the oxen and fastened the plough and the ploughshare to the plough, then I must plough a whole field or more for the whole day.

Consider a typical villein tenant farmer at the time of Domesday Book who holds a virgate of land. For him to plough ten or fifteen acres of his land in February in order to sow a summer cereal crop would likely take - factoring in Sundays - about two weeks of long days of labour. A modern industrialised farmer can plough that same land in under two days.

In a similar vein, a modern combine harvester can harvest around twenty to twenty-five acres per hour, combining reaping, threshing, gathering and winnowing the crops. That would complete the harvesting process for almost the entire land-holding of a typical tenant household in only one hour's work. For a pre-industrialised farm, harvest was the most labour-intensive part of the year and often required the whole household and hired labourers, usually requiring several weeks of hard physical work to manually reap the crops from the field, and then gathering, threshing and winnowing the cereal crops in separate processes. Likewise, a modern crop sprayer can cover as many as 53 acres per hour, spraying modern herbicides or pesticides to remove weeds and other pests. For the pre-industrial farmer, however, the summer months before harvest were a constant process of patrolling their field strips to remove weeds, check for pests and infestations, and to scare away birds and wildlife.

In addition to the automation issue presented by u/BRIStoneman, we would also need to factor several other important issues that shaped the professional makeup of the society.

The crop yields, as you correctly assumed, play an important role. Three-field crop rotation, usage of a modern harness and a heavy plow, and a slightly increased vegetation period due to the temperature increase in Western Europe allowed the farmers in 12th-14th century to collect, on average, 8.5 quintals of grain from one hectare (0.85 t/ha) or roughly 19 bushels per acre (I'm assuming 38 lbs oats bushel), calculated as an average of the four most common grains, i.e. rye, barley, oats and wheat, although one needs to deduct the inevitable spillage and certain amount grain stored for sowing next year. The latter reserve was different in various areas, being heavily dependent on the quality of soil and climate that influenced an average number of grains obtained from a single grain sown, ranging from 13% in Netherlands to 20% in Germany, Poland or England to 33% in Spain, Russia or Lithuania. So, a necessity to reserve 1/4th or even 1/3rd of the entire crop to secure adequate harvest next year heavily influenced the professional composition of the local society.

In comparison, modern agriculture give yields an order of magnitude higher, with e.g. average yield of wheat in 2019 reaching 4.8 t/ha in Ukraine and 8.8 t/ha in Germany with the average amount of sown grain being equal to 180 kg/ha (3,75%). Add to this the new crops introduced to Europe through Columbian exchange that heavily boosted the agriculture. This applies especially to potatoes that allow the yield of 36 t/ha in Germany or England and 49 t/ha in USA, and maize, that yield 100-100 q/ha in USA, Spain or Italy. As you can see, these numbers dwarf the average production capacity of the medieval agriculture.

Please note that the issue here is not limited to Middle Ages but rather to pre-industrial countries. Even though the the number of people working in agriculture in England dropped to 22-25% in mid-19th century, according to the Russian Imperial Census of 1897, rural agricultural workers still formed roughly 75% of the entire population of the Empire (that's more or less the same amount as one estimated by Wrigley in reference to the agriculture in England in late 15th century). In addition, this percentage of agricultural workers was also as high as 53% in 1921 Poland. For comparison, 2020 average for the European Union countries was 5,7%, with largely agricultural countries, such as Poland or Ukraine showing as high percentage as 9-10%, while Germany or Belgium reports less than 2% of the total population employed in agriculture.

Last but not least, the issue of transportation. Without a well-developed networks of modern roads and railways, ubiquitous cars allowing for quick transportation of products, and refrigeration/air conditioning that facilitated storage (this applies to products other than grain), agricultural products must have been heavily localized to maintain the relatively stability of the delivery arrangements, calling for a well-developed agriculture wherever people lived.

Clark, G., Yields Per Acre in English Agriculture, 1250-1860: Evidence from Labour Inputs, in: The Economic History Review New Series, vol. 44, no. 3 (Aug 1991), pp. 445-460.

Mazoyer, M., Roudart, L., A History of World Agriculture: From the Neolithic Age to the Current Crisis, Earthscan, London 2006.

Slicher van Bath, B.H., Yield rations 810-1820, Wageningen 1963.