It also depends on the definition of the "Vikings" here, but some (or I'd say rather the majority) of the population did not traveled abroad or went raiding. While more can always be said, the following previous posts might be useful to satisfy OP's curiosity:

• During the "Viking Age", how common was it for Danes, Swedes and Norwegians to become vikingr? Was this something many people did, or just a small group? What was the social class of the people who went? How did normal farmers in their society see them?: answered by /u/textandtrowel and me
• Did the Vikings that settled Iceland also kept the tradition of raiding?

Some issues to be taken into consideration are:

• It is rather the minority of the population (mainly local elites and their military retinues, though join ownership of the ship can certainly be confirmed) that enjoyed direct access to the ownership of the ship, especially those were specialized in raiding activity.
• To give an example, while they could be traded as goods and sometimes perhaps told to engage in traveling on board (as a non-military job like a cook), the unfree thralls that constituted 20-35% of the total population in Viking Age Scandinavia probably (edited): didn't in general have much opportunity in joining in the expedition.