In short, the structural limitation of the Viking ship was the biggest deciding factor, I suppose.
The ocean-proofing Viking longship owes much of its strength to its flexible framing structure, that is to say, lap-staking (clinker-built) planks and the keel as the ship's 'spine', rather than the ribs and cross-timbers (though they are also important). While the Viking ship was built to turn the power of wave aside by bending the flexible ship body to some extent, it still requires the 'spine', i.e. the keel. The keel should have been made from a good log of the single tree (otherwise, the ship would easily be split into two or more!). Stem and Stern-parts, attached to the keel, also should be carved from a single peace of timber. Thus, the maximum length of the local tree trunk (in Scandinavia) largely determines the size of the Viking ship.
The ocean-proof capacity (as well as part of the swiftness) of the Viking ship was trade-off for its dependency on this structural feature, so if the Vikings used larger galleys instead of their own ships, they would have lost much of their strong point in their voyage and raids as well.
References: