Most soldiers in a medieval army were not knights, nor would they be as heavily armoured as knights. Swords are very good against people without armour.

However, your observation is correct, swords are not the most effective weapons against full plate armour. Weapons like the poleaxes, halberds, or war hammers would be more ideal. Polearms flourished in the late middle ages, in part due to their usefulness against armour.

Of course, you asked about swords, not poleaxes. Swords weren't ideal against full plate armour, but they could be used. A common method of wielding them was to "grip up" on the blade, actually holding it with the hands (which were protected from being cut by gauntlets). With this, a knight could use the hilt and crossbar of the sword in a similar way that one would use a hammer - to bash the opponent, put simply - or attempt to drive home the point of the sword into one of those elusive openings in the opponent's armour.

This would be easier, of course, if the opponent was on the ground, rather than standing and fighting back. A knight might seek to unsettle his opponent's balance (akin to wrestling) and bring him to the ground, and then stab through one of the weak points in his armour, as shown in this drawing, or this one (albeit with daggers, not swords.) These drawings are by Hans Talhoffer, and most of them are from [this fencing manual](http://wiktenauer.com/wiki/Talhoffer_Fechtbuch_(MS_XIX.17-3%29), though I've also used images from [this one](http://wiktenauer.com/wiki/Talhoffer_Fechtbuch_(MS_Thott.290.2º%29) as well.

So, rather that the elaborate duels we're used to seeing in films, a swordfight between knights would more likely be a more brutal affair - get your opponent on the ground, and then deliver a killing blow.