Well leather armor can be a lot of different things from slightly thicker leather clothes to layers of boiled leather clued or sewn together and coated with mineral mixed glue to create something very hard indeed. The famous german pickelhaube is actually made from boiled leather, not metal. Shields were commonly coated with it, horse armor made from it, even tournament armor for knights was sometimes made form boiled leather rather than steel. Why? It's just so much lighter than steel. It is also a lot cheaper than steel.

So how effective is it? Well, the techniques for making boiled leather armor have thankfully not been lost to time even though they have been in use for at least 2500 years. Modern reenactments and test are a dime a dozen. You can find reasonably historically accurate field tests of how effective leather armor was on youtube. Then answer of course comes down to the armor and the situation.

It is also important to remember that leather armor was usually not the only thing you would wear. Most commonly a padded armor called a gambeson would be worn under the armor. It is also important to remember that a well crafted armor is not uniformly thick, it is usually thickest where it needs to be, that is, protecting your vital organs. Piercing two layers of boiled leather with a coating worn over a gambeson is surprisingly hard. A powerful warbow with a 110+ lb draw will struggle even at distances as short as a dozen paces.

More importantly, even if it does pierce it usually doesn't pierce very deep. At short range a warbow will skewer a human like a shish kebab. By that I mean it will easily penetrate a foot or more. Under two layers of boiled leather and a gambeson it might not pierce at all or pierce at most an inch or so. That is the difference between life and death.

What about a sword or an axe though? Well, again, it comes down to the sword, the strike and the type of armor but we can fairly safely conclude through modern testing that piercing boiled leather with a reasonably light one handed sword is almost impossible. Cutting through it just isn't going to to it. A thrust might do it if the sword has a very tapered tip or if there is a great amount of force behind it but you won't slice through boiled leather. Even a very heavy slicing weapon like a dane axe swung with both hands is not particularly likely to pierce boiled leather (the dane axe, contrary to popular belief, is meant to slice, not chop). Added to this a leather armor, helmets in partcular, were quite often reinforced with metal strips or plates to provide additional protection against slicing weapons.

So how effective was it really? There is no way to answer this. Boiled leather over padded armor was a good kind of armor, comparatively cheap and light while offering relatively good protection. The fact that it has remained in use for quite literally thousands of years is a testament in itself to how effective it was. A soldier would generally speaking wear the best protection he could afford and for a very long period for a lot of soldiers, that was boiled leather.

EDIT: Don't miss the response below by /u/waritter

To my knowledge the only published academic account of leather armour's protective abilities is in Alan Williams' the Knight and the Blast Furnace, and that is somewhat limited because his focus was on plate armour, not leather armour. To test armour, he set up an impact tester - basically an apparatus with a dropping weight of a given mass and pointed instrument at the end, shaped to simulate the weapon impact being tested - a blade for axes and swords, and various points for arrows and lances. This apparatus can be used to set up an exact energy at impact, so the resistance of armour to attack can be tested.

He tested two types of leather - hardened 'cuir brulee' shaped as a kind of dish and 5mm thick (a little less than a quart of an inch, for us Americans), a softer 'buff leather' of an unspecified form and thickness, probably flat. The buff leather was penetrated by the simulated blade at 90 joules of energy - less than a pollaxe stroke, signifcantly more than a sword swing. It was penetrated by the simulated arrowhead at only 30 joules - most estimates for English warbows range between 60 and 100 joules of energy (as you can tell, they vary quite a bit!), which is much more. The cuir brulee faired better against a simulated blade - it was defeated at 90 Joules, but was defeated by a simulated lance head at 30 Joules. I have seen some other tests - generally performed live firing bows at targets - but they have other flaws, or haven not been published or peer reviewed.

Though these tests are limited, they tell us a bit. Namely, compared to steel armour formed into mail or plates, they were not extremely protective, but were useful against things like sword slashes. Indeed it is less protective compared to a layered jack made of 30 layers of linen, which resists a simulated blade impact of 200 Joules In Williams' same tests. One caveat to keep in mind is that the cuir brulee used was 5mm thick - it is possible that originals were thicker - unfortunately the leather horse armour listed in the "Armoured Horse in Europe" catalog lists no thickness.

This makes sense given how leather was used in European armour - it was a supplement in armours for men (as hardened leather over mail in the 13th/14th century, as a flexible deerskin outer layer of layered cloth jacks in the 15th century), an armour for horses, and protective equipment for combat sports like the club tournament, where impacts had to be blunted but penetration was not a serious concern.

After the middle ages, in the early modern period, buff leather was used as armour to protect against sword slashes, and based on these tests it would be pretty good at that.