While round musket balls had a tendency to lose both accuracy and speed (and thus penetrative power) quickly over long distances, at short distances they easily penetrated the heaviest of armour due to the penetrative power in the combination of velocity and the high weight of the large caliber lead musket ball.
Armour was worn, but most often only by men that would also expect to fight in melee, where armour would aid even further. Pikemen and heavy cavalry continued to wear armour to a greater or lesser degree throughout the era of early black powder, pike-and-shotte and early linear warfare.
One also needs to remember that armour is expensive, cumbersome, heavy and restrictive. Due to the inaccurate nature of musketfire, most tactical doctrines of the time called for dicisplined mass fire at close distances to create the most amount of damage, both physical and psychological.
Armour that could protect against balls from long arms (ie muskets) were too heavy to be of practical use. Breatsplates (or cuirasses) were often made thick enough to resist pistol bullets at short range, and other armour only thick enough to protect against sword slashes and stabs.
For example, Swedish and Finnish cavalry during the 30 years' war were instructed to not fire pistols until they could actually physically touch the armour of their enemy with the barrel of the pistol to maximise the chances of penetration of the armour of the Imperial Cuirassiers.
Likewise, the Swedish cavalry were equipped with cuirasses up to the early part of the Great Nordic War, and these were proof-tested by having a pistol shot at them at close distance - a cavalryman was instructed to not accept a cuirass that did not have a dent from a test. Here's an example from the late 17th century. Note the dent in the middle. Here's another example, note the dent again.
These weigh between 6,5 and 7kg, for only the breast (no the back part, which was usually thinner), and is only proof for pistol bullets, not musket balls. Imagine the weight of armour that would protect all vital parts and protect against musket balls at close range!
