The short answer to this is that it is not known why it is square. Like Orwell's war with Eastasia, the flag of Switzerland has always been square. Especially since the law that lead to the adoption of the symbol as the symbol for Switzerland did not make a difference between flag and national coat of arms. It said:

The coat of arms of the federation is, within a red field, an upright white cross, whose [four] arms of equal length are one and a sixth times as long as they are wide.

Until a recent law in 2017 neither the relation between the cross and the flag, nor the square form of the flag were officially signed into law and as stated above, when adopted it served both as coat of arms and flag at the same time. Similarly, the specific shade of red was also not defined until 2017 when Switzerland officially defined Pantone 485 as "Swiss Red".

Like its square shape, the exact origin of the Swiss Cross on Red is also unknown. Various authors point to the battle of Laupern 1339 as the first use of the cross for troops of the Old Confederacy (The Old Swiss Confederacy refers to the federation of cantons and cities that existed on the territory of Switzerland until Napoleon conquered the territory and created the Helvetic Republic), the evidence for which comes from the Berner Chronicle written in 1420.

Wherever it came from, wider use of the white cross on red ground as a marker of Swiss troops is attested from the mid-16th century forward when Swiss mercenaries used it as their sign in battle and it was also adopted as the official coat of arms of the Canton of Schywz, which was one of the original members of the Old Confederacy and which inspired the modern name of Switzerland in German, Schweiz.

Anyway. like many things, the flag confusion went out of the window with the invasion of Napoleon, who was responsible for, among other things, spreading the use of national flags in general, preferably with three colors. And so it was in the case of Switzerland. With the end of the old confederacy and the start of the Napoleonic Helvetic Republic came this flag, which represented the new order and which in some cases could be found in both square and rectangular versions.

The end of the Napoleonic order with the Vienna Congress in 1815 brought – again – a new order in terms of flags. For Switzerland specifically the use of the Swiss Cross on Red Ground as a popular military flag. Referring back to the older symbol used by Swiss mercenaries and the Canton of Schwyz, General Niklaus Franz von Bachmann first introduced the symbol in form of an armband as the symbol all Swiss troops would use in order to distinguish themselves. This was then further expnaded in the 1830s and 1840s when General Guillaume-Henri Dufour introduced the symbol as a flag for all Swiss army units in addition to the flags and symbols of their cantons.

From the 1830s onward, many Swiss politicians and reformers had also been pushing for a new organization of the confederation and had used that symbol as virtually one of the very few ones available to them (aside the Napoleonic HR flag, which they didn't like for understandable reasons) as a symbol of unification reforms. When this finally came true in 1848 and Switzerland got a constitution that lead to them founding the federal state of Switzerland, they did not yet adopt a national flag, but at that time only a military flag: A White Cross on a red field.

This would then finally become both the coat of arms and the flag of the federal state of the Swiss Confederation in 1889 and by that time the square shape had already had a long tradition, even when used as a military flag – a tradition that held true until officially singed into law in 2017.

Now, it is possible to speculate that the square shape of the flag was related to its traditional use as a military flag/banner. However, according to most primary sources, neither Bachmann nor Dufour nor any modern military source mentions whether the flag was supposed to be square or rectangular. It was just far more often used in a square shape and so the unsatisfying answer to this question is simply: The Swiss flag is square because the Swiss say it's always been square.

As for the Vatican Flag: The only relation is that both flags gained widespread use and popularity because of the Napoleonic wars – a real vexiological turning point as it turns out – and is reminiscent of the flag previously used by the Church State in Italy, which was also square, only with red and yellow as its colors. As with Switzerland the answer concerning the square shape is tradition. It's been square before so it is square now.

With flags as with, say, the DIN A4 paper size standard, the international wide-spread adoption of a sort of standard just happened really, with no law or event prescribing that flags are supposed to be square or standard paper be 210 × 297 millimeters in size. People sort of, kind of, just did it because others did it and so it spread with only few outliers in both cases (looking at you, US letter size paper and Nepalese Flag).

Sources:

• Louis Mühlemann: Wappen und Fahnen der Schweiz. Zürich 1980.

• Peter F. Kopp: Schweizerkreuz. In: Historisches Lexikon der Schweiz.