First off is my understanding correct that they couldn't really do damage to one another with standard melee weapons?

It's better to say that they had much more difficulty doing damage to each other with weapons.

If so, I imagine they didn't just trade ineffectual sword strikes. What was the goal in this situation?

Armour has gaps, and some parts of armours are thinner and less protective. Hans Talhoffer illustrated the kind of things that could happen in an armoured judicial duel in his mid-15th century fightbook. He shows some of the tactics and techniques that could be used, including swords (despite the armour being "swordproof").

1. The first step is preparation. Apart from having done substantial training before this, the preparation should also include prayer and bringing a coffin: https://commons.wikimedia.org/wiki/File:Ms.XIX.17-3_02r.png (for some judicial duels, the loser would be executed (since he was guilty) if he survived the duel)

2. The duel could conventionally open with the throwing of spears. This could potentially penetrate a helmet visor, or thin sections of the armour like arm and leg plates (but the curvature of these would make it likely to glance off): https://commons.wikimedia.org/wiki/File:Ms.XIX.17-3_11v.png

3. Techniques with the sword include the "murder strike" or "Mordschlag": https://commons.wikimedia.org/wiki/File:Ms.XIX.17-3_08v.png where the sword is held by the blade and used as a club in an attempt to stun the opponent.

4. The sword can be held with one hand on the hilt and one mid-blade ("half-swording"), to try to drive the point into gaps in the opponent's armour, and to assist in wrestling: https://commons.wikimedia.org/wiki/File:Ms.XIX.17-3_14v.png

5. These are quite complete armours being worn, with the backs of the legs covered, and mail protecting most of the other gaps. However, even with such complete armour, the palms of the hands are often unarmoured: https://www.wiktenauer.com/wiki/File:MS_78.A.15_22r.jpg

6. The sword can also be used to trip the opponent: https://www.wiktenauer.com/wiki/File:MS_78.A.15_21r.jpg

7. However, this part of the duel, with swords, often failed to bring it to a conclusion, and the fighters would resort to grappling: https://commons.wikimedia.org/wiki/File:Ms.XIX.17-3_19r.png

8. In grappling, the key anti-armour weapon was the dagger: https://commons.wikimedia.org/wiki/File:Ms.XIX.17-3_18v.png

9. ... which could be driven into gaps while grappling: https://commons.wikimedia.org/wiki/File:Ms.XIX.17-3_20v.png

10. With sufficient control of the opponent in grappling, one can make larger gaps in their armour, bringing the fight to a quick finish: https://commons.wikimedia.org/wiki/File:Ms.XIX.17-3_21r.png

11. .. at which point the loser can make use of his coffin: https://commons.wikimedia.org/wiki/File:Ms.XIX.17-3_22v.png

12. and the winner can thank God: https://commons.wikimedia.org/wiki/File:Ms.XIX.17-3_23r.png

Was there a difference in a large battle versus 2 knights duelling?

On the battlefield, one would seek to avoid one-on-one. If an enemy knight was foolish or unlucky enough, it was much better to fight them 2 or 3 (or more) against one, leading to quickly wrestling them down, at which point they will surrender or die. Again, in this situation, the main anti-armour weapon might be the dagger.

Battlefield armour were often less protective than duelling (and tournament) armours. Equestrian armours often had no protection for the buttocks or the back and inside of the thighs, helmet visors might be opened for better vision, hearing, and breathing.

Mounted knights could be badly injured or even killed if their horse was brought down.

The weapons on the battlefield also included bows and crossbows, and sometimes guns, javelins (which are potentially good anti-armour weapons), and fire-bombs.

Also, were heavy hammers and maces able to do damage where swords could not? If that's true then why did they not become the dominant weapon?

At the time, two-handed polearms (pike/spear, halberd, bill, etc.) were the dominant hand-to-hand weapons on the battlefield. They didn't outnumber swords, since pikemen, billmen, and halberdiers usually carried swords as sidearms, but they saw more use in battles.

While heavy one-handed hammers, axes, and maces (often about 1.3-1.5kg, significantly heavier than the average one-handed sword, and with the mass concentrated near the end) were more likely to crush armour, penetrate armour (using spikes on hammers and axes), and cause damage through armour than swords, they were not as deadly as their bigger two-handed relatives. The thickest and most protective parts of armour were still quite resistant to such hammers/axes/maces (e.g., Henry V was hit by an axe or mace in the face at Agincourt, but only suffered minor damage to his helmet). A cavalryman could, and often did, carry such weapons as a supplementary sidearm (along with a sword), using a lance or pistol as his main weapon. Heavy weapons like that have two disadvantages compared to swords: they are heavier and more unwieldy, and therefore slower, and can be less effective against unarmoured opponents. A big advantage of such weapons compared to swords was that they were less likely to be damaged when hitting a fully-armoured opponent (this same advantage also applied to lighter maces, hammers, and axes).

The most effective anti-armour hand-to-hand weapons (other than the dagger when grappling) appear to have been polearms like the halberd, bill, and pollaxe. These were often about 3kg in total mass, and the long haft allowed the head to be swung through a long arc, delivering a strike with kinetic energy aplenty. Spikes on such polearms should have been able to penetrate the thinner parts of most armours (often only about 0.8-1mm thick), and could threaten helmets and especially helmet visors. Hitting with a blade or hammer instead of a spike would be likely to crumple thin armour and could concuss through a helmet. In the Battle of Flodden (1513), the front ranks of the Scottish pikemen were well enough armoured so that English archery had little effect. English bills, however, dealt severely with them once their formation started to lose cohesion. The Scottish king, James IV, killed on the battlefield, only suffered wounds in gaps in his armour (face, neck, wrists), so armour could still provide protection against such "killer" polearms. On the other hand, Charles the Bold, Duke of Burgundy, was killed in the Battle of Nancy (1477); his boy was found with a halberd wound to the head, and spear wounds to his legs and abdomen, despite his excellent armour.