I first need to address a misconception in the question itself, as you seem to be confusing 19th century and 20th century trends.

While it is quite true in the US that farm consolidation (spurred by technology and other factors) caused a drop of number of farms (and increase in average farm size) the actual flip didn't happen until the 20th century -- you're probably thinking of a chart like this one. When someone left their family farm, it was more likely due to transportation -- that is, accessibility of urban areas. Also, on average, it was not farmers leaving agricultural work, but the children of farmers.

Rural numbers in the US grew during the 19th century, just at a slower rate than urban areas. The vast growth of immigrants was focused in urban areas. So an average factory worker wasn't necessarily dragged from rural to urban life.

The other issue is the term "industrial revolution" itself, which modern scholars have found dubious at best, an absolute misnomer at worst. There are too many overlapping events, too many starts and stops, and a good number of the associated technological innovations happened outside the classical assigned time (roughly 1760 to 1843). In more popular texts (and your question) it can get shifted to the entire 19th century. For our purposes let's just avoid the term altogether and think of a.) what agricultural life was like in the late 18th century b.) what urban factory work was like generally in the early 19th century; was it unrelentingly grim? how did they survive, mentally and physically?

What was agricultural life like?

This is precisely the sort of question that is historically elusive, as by its nature there was no set timetable, and hours could vary by location or cultural factors. (Chesapeake had a tradition in agricultural labor of four hours off mid-day Saturday, which did not necessarily hold farther north.) The very individualized nature of farming meant we have trouble spinning any kind of average experience. Just keep in mind that not everyone owned a farm; the percentage of operators in amongst free household heads in 1774 New England was at 43.9%, with "professions / commerce / crafts" at 11%, "no occupation but some wealth" at 16.7%, and laborers at 28.4%. (The Southern colonies had rather different percentages, with only 1.9% as menial laborers, but also much more reliance on slavery.)

We can pull specific examples: Francis Pepper of Colonial Springfield, Massachusetts started at 4:30 tending livestock, with a half hour break for breakfast, four hours of work, half hour break, one and a half hours work, one hour dinner break, three hours work, one hour break, followed by bed. If you do the math, that's 10.5 hours per day. This isn't too far from a 19th century factory in terms of hours (although he does get to choose his own pace and variety of work).

Or, Matthew Patten a farmer who kept a diary starting in 1754, and had a farm entirely tended to by him and his family. He had what you might think of as a more casual agricultural schedule, spending (on average) only 8 hours per day overall with work (planting seeds, moving firewood, etc.), and managing in his leisure time to go fishing, hunting, and attend community meetings. (These hours were spread out, though -- some months had lots of work, some had none at all.) This diary manages to continently fall just a bit before the start of factories in America, the first (in 1790) being a textile manifesting factory brought over by Samuel Slater from England.

What was factory work like? Was it unrelentingly grim? How did they survive, mentally and physically?

Again, it's difficult to consider an "average" factory as they varied by owner and nature. A cotton mill only hiring girls from 8 to 12 is rather different from one employing adults.

It is hoped that those citizens having a knowledge of families, having children destitute of employment, will do an act of public benefit by directing them to the institution.

-- Advertisement asking for children 8 to 12 to work in a cotton mill, Baltimore Federal Gazette, 1802

We have more written evidence from England than America on this; one factory owner bemoaned having trouble with

... training human beings to renounce their desultory habits of work, and identify themselves with the unvarying regularity of the complex automaton.

another noted

I found the utmost distaste on the part of the men, to any regular hours or regular habits ... The men themselves were considerably dissatisfied, because they could not go in our out as they pleased, and have what holidays they pleased and go on just as they had used to do...

Some fines in early cotton mills (tending to be but not always children) included

Idleness & looking thro' window

Calling thro' window to some Soldiers

Riotous behavior in room

Riding on each other's back

Dancing in room

Going out of the room in which she works to abuse the hands in another room

Making a noise when order'not

Terrifying S. Pearson with her ugly face

Sending for ale into the room &c [etcetera]

So there were, of course, acts of defiance during work (although outright dismissal was the most common tactic used to counter) including "celebrating St. Monday" after a heavy night drinking the day before.

Even at 70 hours a week it was possible to get in some time, however small; larger sporting events like foot races and horse races could attract large crowds in the early 1800s US (in the tens of thousands) and this included the factory-working class. The moments were definitely sparse, and I'm afraid I can't quite fully answer your underlying notion of "what would it be like inside their head?" People still lived, they still celebrated, they still drank, they still danced, maybe just a little less than they would like. I think the important point is, for someone where the confines of a particular job were too much, where they were not someone who wanted to prove their ambitions to an employer with industriousness, they were perfectly willing to be defiant and carve out time for themselves.

...

Applebaum, H. A. (1996). Colonial Americans at Work. United Kingdom: University Press of America.

Atack, J., M. Haines, and R. Margo (2009): “Did railroads induce or follow economic growth? Urbanization and population growth in the American Midwest, 1850-60,” NBER Working Paper, 14640.

Carr, L. (1992). Emigration and the Standard of Living: The Seventeenth Century Chesapeake. The Journal of Economic History, 52(2), 271-291. doi:10.1017/S0022050700010731

Cunningham, H. (2016). Leisure in the Industrial Revolution: C. 1780-c. 1880. Routledge.

Gutman, H. (1973). Work, Culture, and Society in Industrializing America, 1815-1919. The American Historical Review, 78(3), 531-588. doi:10.2307/1847655

Hopkins, E. (1982). Working Hours and Conditions during the Industrial Revolution: A Re-Appraisal. The Economic History Review, 35(1), new series, 52-66. doi:10.2307/2595103

Jenkins, J., & Pigram, J. (Eds.). (2004). Encyclopedia of leisure and outdoor recreation. Routledge.

Lindert, P. H., & Williamson, J. G. (2016). American colonial incomes, 1650–1774. The Economic History Review, 69(1), 54-77.

Mokyr, J. (2018). The British Industrial Revolution: An Economic Perspective. United Kingdom: Taylor & Francis.

Pollard, S. (1963). Factory Discipline in the Industrial Revolution. The Economic History Review, 16(2), new series, 254-271. doi:10.2307/2598639

Schuman, M. (2017). History of child labor in the United States—part 1: little children working. Monthly Labor Review.

Smith, B. (1988). Poverty and Economic Marginality in Eighteenth-Century America. Proceedings of the American Philosophical Society, 132(1), 85-118. Retrieved July 23, 2021, from http://www.jstor.org/stable/3143826

Voth, H. (1998). Time and Work in Eighteenth-Century London. The Journal of Economic History, 58(1), 29-58. Retrieved July 23, 2021, from http://www.jstor.org/stable/2566252

You'd be interested in this previous post, Is there any historian that analyzes the Industrial Revolution from a positive point of view? with a good answer by u/AlviseFalier, and u/teabeforetherain's suggestion of Emma Griffin's book Liberty's Dawn: A People's History of the British Industrial Revolution for a historian's argument that working class people had a better life in cities during this period than most people would think.