In fact, Americans in the 1860s tended to be unusually tall, and that included soldiers. One study found the average American soldier in 1860 was 5'7" tall (170.6 cm). That's an inch or two taller than major European countries — the British and Dutch averaged around 5 feet and 5.5 inches; the French were shorter still.
Another study looking solely at American soldiers similarly found the average male soldier in 1864 was just over 5'7" tall; the average soldier in World War II was 5'8.4" tall, just over an inch higher.
Studies looking at 19th Century male height almost exclusively look at soldiers, because that was often the only consistent way of collecting data. But these data should be looked at carefully — because European armies, at least, usually had minimum height requirements for soldiers, and surprisingly large numbers of European conscripts flunked that test.
Looking at France, my specialty, in the 1820s "about a quarter of boys liable for conscription were refused as being below the minimum height to handle a musket (only just over 5 feet)," per historian Robert Tombs. The shortest, most malnourished soldiers came not from the teeming large cities but from the poorest rural areas. Eugen Weber gives some specific examples from the department of Lot in southwestern France; in one district, the 1830 draft class consisted of 1,051 draftees, of which 212 (one-fifth) were rejected. Of those, the cited reason was stuntedness (défaut de taille) for 120. Consistently, Weber notes, up to half of all people rejected for military service were rejected for shortness. (Many of the others had various physical deformities or ailments, which are also associated with malnourishment. "In 1860 one rural area near Grenoble had this to show to the army recruiters: goiter, 140; deaf or dumb, 13; lame, 13; myopia, 36; tapeworm, 19; scabies, 1; skin maladies, 86; scrofula, 15; epilepsy, 2; general weakness, 197; hunchback, 29; bone distortion, 2. In a total of 1,000, 553 men were disqualified, only 447 found fit for service.")
Alain Corbin's case study of a 19th Century French peasant (born 1798, died 1876) notes that he was 5'5" tall and that "for this time and place, in other words, he was a tall man."
One 1849 discussion of "men remarkable by their height" to serve in elite French units "makes clear that they are between 5'6" and 5'8" tall," Weber writes. These elite, unusually tall French soldiers would have been average or even short in the Union Army!
By 1860, the situation in France had improved somewhat — Emile Durkheim found "the number of conscripts rejected for height dropped from 92.8 per 1,000 in 1831 to 59.4 per 1,000 in 1860," but that's still more than 1 in 20 men being less than around 5' tall. Today 1 in 297 adult Frenchmen would fail that test. (This reflects improvements in agriculture and transportation during the 1850s.) Even the soldiers who were healthy enough to be admitted into the army often required "a few months of solid regular feeding, including 340 grams of meat a day, before they could cope with the physical demands of service."
Unlike in Europe, the Union Army didn't have codified height minimums, but rather a subjective judgment by an examining doctor. One American officer summarized the situation in 1875:
Weight is not a regulated quality in any code of laws governing the enlistment of recruits. The circumference of chest thought to be indispensable as an accompaniment to certain degrees of stature, is carefully laid down in the English regulations, but weight is not even mentioned. It is presumed that the matter is left to the discretion of the examining surgeon, with whom the decision as to the other qualities named might, it is thought, be also left with advantage. A due proportion in the weight is quite as essential in the soldier as a well-formed chest, and is of greater importance than lofty stature. In former times, when it was necessary to make use of a ramrod in loading a musket, men of a certain height were absolutely necessary for the service; but in these days of breech-loading arms, a man from 5 feet to 5 feet 4 inches in stature, and well proportioned in build and weight is, ceteris paribus, as serviceable a soldier as can be desired.
Americans were shorter then than they are now, but if you're comparing them to other Western countries, Union soldiers were remarkable not for being short but for being tall.
EDIT: I've seen nearly half a dozen replies now asking questions about my use of inches and feet. To preempt more: Those are standard, modern inches and feet. In most cases the raw height data was given in centimeters and I translated it into modern feet and inches because the question was asked using those units. No, the differences here are not explained by the confusion about different units called the "foot" that is commonly cited as a reason for the (largely false) belief in Napoleon's shortness. Thanks.
Sources
I have a related question I would like to tack on if OP doesn't mind. A popular movie trope about this time period is immigrants, in particular the Irish, going straight from the boat to the Army, while I doubt it was this straightforward, what percent of the soldiers fighting in the Civil War were recent immigrants and were there any trends or themes in how they chose to enlist/were drafted?
Follow up question. How does that height and weight compare to the average American male and average European male of the time?