Let's say a twenty year old man who is about 5'8" tall and about 180 pounds on the lower end of 'in-shape' goes back to Rome under Augustus. He's found in a forest in the Apennine Mountains and taken in to a rich household in Rome as an slave. Based on physiology, is this about the average size of a Roman? Could they be able to handle the workload of daily works at an Olive orchard? How about fighting Germanic raiders? Would they be able to win an Arm-wrestling match? What other tasks might he have trouble with besides learning the language?
The Logistics of the Roman Army at War: 264 B.C. - A.D. 235 by Jonathan Roth provides some height and weight estimates.
Depending on exact time and place your theoretical time traveller would be 4" or so taller than the average roman. And he would meet the height requirements for military service. But he might not be allowed into some units. Soldiers of a first cohort had to be 6 roman feet tall (which is about 5'10" in modern US measurements). So he would be right on the edge.
He would be heavier than most people his size. And he would probably be in much worse shape, especially compared to trained soldiers. Legionaries had physical fitness requirements. Vegetius mentions that in training recruits were expected to complete marches of 20 miles with a "pack" of 60 lbs in 5 hours while keeping in proper ranks in a marching formation. They also had a variety of other physical "exercises" including running, jumping, wrestling, and where possible swimming. And of course they had weapon training. And weapon training was done with wooden replicas twice the weight of normal weapons in order to build muscle strength. Compared to a soldier your time traveller would get slaughtered by a German raider.
He would also probably be unprepared for difficulty of manual labor that any type of agricultural work would require. But I expect he could adapt within a few months. Modern militaries train unprepared raw recruits all the time.
New slaves and new soldiers were being taken in all the time and they came from everywhere in the Empire, so your theoretical modern man would be considered rather tall (but not freakishly) and maybe a little bit fat as calories were not in surplus even for more freedmen or citizens. He would definitely be physically soft, but so would other recruits/new slaves, especially slaves, who were often coming off long sea voyages from wherever they'd been captured, during which they had no real exercise and were probably undernourished.
Tallfatso might get a few extra beatings until he was in shape, but he could get there quickly enough. People on The Biggest Loser would lose a lot more if whips were involved.
Language also wouldn't be much of a problem as the Empire contained hundreds of languages and no one really expected to understand everyone else. For many slaves, nuanced conversation wasn't a job requirement anyway. A fresh slave from Gaul didn't speak much Egyptian, for example, but he might find himself working there.
The much larger problem would be adjusting mentally. Rome was a rough and, to many modern perspectives, absolutely terrible place. Soldiers were subject to beatings or executions at the whim of their officers. Slaves truly were objects and their owners could do literally whatever they wanted to them. They could work them to death, rape them, disfigure them or kill them outright without any fear of punishment from the state. Indeed, there were "outside contractors" whose business was the torture and execution. Owners would contact these firms to have them crucify an irritating slave the way we might call a plumber to have our sink repaired.
Any modern person transported back to Augustan Rome would have to accept these new ethical/moral standards quickly, or risk running afoul of them.
Then the food and (lack of) sanitation would probably kill Tallfatso. Cholera, polio, bacteria. Just wreck him completely. Assuming he was American or similar. Someone from the modern third world would probably do a lot better in all respects.