Cannons need the space to recoil, when you fire them they role back on their tackle sharply and with a lot of force. Larger bore guns are going to have a much longer recoil meaning you need to set aside much more space to be able to rock back on that tackle. Attempting to stop a cannon doing that would be dangerous and could result in the ship being damaged by the force of the recoil not being softened by the reduced moments involved in allowing it to recoil properly. Given ships are longer than they are wide its therefore much much easier to be able to design a ship with more smaller guns than less bigger ones. Potentially you could have experimented with place them in some kind of alternating pattern to be able to give them more space to do this but i suspect you'd end up net losing weight of broadside. Additionally you have to consider that while potentially having less gun teams might at least some what reduce the necessary number of gun teams but you are still likely to need a larger team to load a cannon of a larger size with the same speed. The ball is heavier meaning its harder and more tiring to pick them up and move them around with the same speed and the weight of the gun once its been reloaded has to rolled back forwards on the tackle and moved to aim it. This could also combine with some problems with stability should a cannon not be fully emplaced given as the a ship rolled with the waves or heeled in the wind if the guns weren't bound tightly in place, and as i've mentioned you couldn't do that if they were to be ready to fire, the weight of the guns rolling with the tilt of the deck could potentially be destabilising. This is suggested to be a reason why a number of large ships earlier in the period that had been particularly heavily packed with guns sunk suck as the Mary Rose and the Vasa but this isn't 100% confirmed.

There was something of a solution to this but there was a trade off to it. Carronades, basically much shorter cannons of a relatively high bore diameter. But that shorter length means that the cannon ball has less time to be accelerated in the gun meaning you don't get quite the same range or accuracy out of them as you might a longer gun. People, the Royal Navy and US Navy in particular, played around with using them a fair amount in various capacities. Some very small ships had most or all of their broadside be these carronades, kind of as a tool of self defence so that they could blast a ship that tried to close in at a short range to board them with a weight of broadside that'd had a nasty sting for its size. Some vessels that that were more equipped with offensive action in mind occasionally also had a carronade heavy loadout, sometimes frigates, sometimes smaller vessels like sloops-of-war that were nimble enough to close the distance with a foe such that a vessel of a similar size or smaller would find itself with a very heavy broadside facing it if he quick vessel could close the distance before it suffered significant damage.

Larger vessels sometimes just carried a few and given its relatively short length and for their bore short recoil it was easier to fit them in places longer guns found it difficult to be squeezed in such as a bow chaser just to add a little extra short range fire power in a direction it was difficult to get a gun of any size facing. This was as an alternative to making your chase guns very low bow and fairly long which was ideal if you were involved in a pursuit (or chase if you will) and wanted to try pull a ball or chain through something that'll slow the other ship down. Of course being able to do this at long range increases the chance you'll win a pursuit.

Some of the other things that did occur was during Razees, that is when a naval ship has the number of decks she has reduced to make a faster vessel at the cost of some fire power. Often when this was done with a ship of the line to make it into a large frigate the ship with loose some of her smaller guns reducing the total number of guns while keeping her largest guns. While other frigates of a similar kind of profile of guns existed given that these frigates were still as long and wide as a full ship of the line, just with less gun decks, they did tend to have some of the heaviest guns for a frigate about.

The other thing is sometimes a ship would sacrifice some fire power for a few large guns that could fire explosive shells. In early periods these were things like mortars and had to be specially built with strengthened decks to take the force of the recoil of a mortar against the deck at expense and might well carry a smaller profile of normal guns. This kind of vessel was mainly designed with shore bombardment in mind as it was arguably harder to land a good shot on a moving ship, particularly from another ship with a mortar than with a cannon with a lower arch. Latter on as guns that fired more modern kinds of explosive shells were becoming more common in the mid 19th century and a few powers like the US navy converted old frigates into sloops with less guns but added on one or two such powerful guns.