I’m trying to write a fantasy novel in a world that has similar naval/shipbuilding technology to the age of exploration. The characters are tasked with exploring and charting the lands west of a large ocean, but I want the ships to have some kind of capacity to field a large enough amount of infantry to square off against natives and small towns of a similar technology level.
I guess I’m wondering how many people a ship of that age could transport and still be seaworthy far from shore on a long expedition. And, since it’s a fantasy novel, how big a wooden exploration vessel could theoretically be without becoming unlikely or detrimental to the voyage.
This is one of those questions that has multiple answers. I have no idea how theoretically large a wooden ship of that era could be; I'm not a shipwright. The best I can do is to tell you how large the largest wooden ships actually were, and then explain the problems with using them for the role you intend.
I assume you're primarily interested in the late 15th century and 16th century. The largest warships of the early 16th century, heavy 500-ton-plus carracks like the Mary Rose, carried considerable crews. The Mary Rose had a normal complement of ~200 sailors, ~30 gunners, and just under 200 soldiers, but could ship double the number of soldiers on short-ranged expeditions. Henry Grace a Dieu, the Mary Rose's larger cousin (but which we know less about) was around 1000 tons burthen, with a crew of 700-800. The late 16th century galleons that made up the Spanish Armada carried anywhere between 130 and 362 troops on that ill-fated voyage, though it's important to remember that this was for a relatively short-ranged invasion, not a long voyage of exploration.
Very large ships were not often used on voyages of exploration. Columbus' first voyage was made with one nao and two caravels. The nao, Santa Maria, displaced about 150 tons and had a complement of forty men, and the two smaller ships displaced well under 100 tons and carried less than thirty men apiece. Magellan's expedition consisted of four small carracks and a caravel, with an average complement of 54 each.
There are a great many reasons that recommend against the use of very large ships for this purpose. They are hideously expensive, not only to build, but to arm, to crew, and to provision. A large crew, and especially a sizable detachment of soldiers, eat through rations and fresh water stores at a very rapid rate. And what would it gain them? Generally, explorers had no need of scores of cannon or a massive hull that can absorb gunfire. The ships' great size and bulk would be an active disadvantage in some situations. Large carracks tended to be top heavy, which caused issues in heavy seas. Additionally, the heavier a ship is, the lower it rides in the water; the lower it rides in the water, the deeper the water must be. A large ship will run aground in shallow seas that a smaller ship could navigate safely.