I think it's important to clarify the use of the term yeoman, it means a wide variety of things in different countries and at different points in time. The yeoman in colonial America were considered a persecuted class, Jefferson was merely speaking out for their rights. They were often made up of mixed race communities. Many of them white runaways from their colonial masters, many of the early European settlements were little more than prison colonies. With no noble ancestry they lived in small settlements of subsistence farmers mostly in the Appalachian mountains. As America was being founded discussion of whether or not to include these territories included integration of the illegal Yeoman settlements along the frontier. This however would include giving them equal rights and brought up the racial integration question, which obviously the southern states were firmly against.

​

https://msuweb.montclair.edu/~furrg/essays/fieldsideolandrace.html