Problem with the Spartans was that while they had some of the best trained troops in Greece, their society as a whole was pretty much stagnant.

First, very few of the people in Sparta were actually 'proper' Spartans, the vast majority were slaves called Helots, these slaves were a constant threat due to the possibility of revolt, and the fact that the Spartans themselves were pretty much forbidden to do any farming, crafting, or anything other than training for war, meant Helot rebellions were much more damaging than in other city states where citizens were at least somewhat sustainable. This system where the Spartans were full time soldiers and almost all the economic production was left to a slave force that was always in danger of revolting meant that Sparta had to spend a lot of their military just keeping the Helots in line.

Sparta also had big problems with a forever decreasing population, their society wasn't really based on the family unit like most of the other city states (like Athens), but to the state. Males were taken from their families at a young age and lived in a barracks separate from even their wives until about 30, at least. This, as you can imagine did not really help with baby-making T_T. Even if a baby were born there was a chance the elders might decide the baby was unfit and have the child left to die of exposure.

So pretty much: Sparta had very good soldiers because they could train to be soldiers full time, but the way their society was constructed meant that they had to spend most of their military power keeping the helots under control so they could have the food and supplies needed to maintain their full time soldiers. In addition, they also had a very low birth rate, meaning Spartan soldiers were very slow to replace.

Athens didn't need to waste so much manpower keeping their slaves in line, and their birth rate was much higher(because they didn't do that thing where husbands lived apart from their wives) meaning they could absorb losses much better than the Spartans.

The Spartans were a society that was pretty much incapable of growing, all their time, effort, and resources were spent just keeping their society running.

There are a number of reasons.

First off, Athens gained much of its power through naval dominance and diplomacy, a field that Sparta could not match them in for the early portion of their classical history. Before the start of the Persian Wars, Sparta considered itself the leader of the Greek city-states, but as the war continued, Athens took much of the initiative by building a strong navy, winning decisively in the subsequent naval battles, and taking the fight to the Persians in Asia Minor. With this, the Athenians began to Challenge the Spartans for their title as leading city of Greece. The Athenians formed the Delian league, ostensibly to unify the Greeks in their fight against Xerxes, but it was seen by many as an attempt to create an Athenian Empire.

Of course, this did not make the Spartans nor a few other big city-states very happy, and these tensions led to the Peloponnesian War (431-408 404 BCE) between the Delian League (led by Athens) and the Peloponnesian League (led by Sparta).

But even though this was the big eruption of rivalry between the Athenians and the Spartans, circumstance generally made it difficult for the two powers to fight head-to-head. Sparta was a land power (for most of its history). Athens was a naval power. Though the Spartans were able to summon nigh-unbeatable armies to destroy the Athenians, they did not really have the opportunity to use them. One interesting case of this was during the Archidamian War (the first leg of the Peloponnesian War). The Spartans had unsurprisingly succeeded in invading Attica and surrounding Athens. Though this deprived the city of its surrounding farmland, Pericles (an Athenian general at the time) came up with the ingenious idea of building the 'long walls', which basically extended the city walls from Athens to the nearby port of Piraeus in a narrow strip. This was done because Pericles knew that they could get as much food as they needed by sea, and they knew that the Spartans, though formidable, would not be able to challenge them here. Moreover, Pericles warned against facing the Spartans in open battle, because he knew they were just too strong and too numerous. At this point, Athens was able to keep up with its rival by existing in a way that the Spartans, for all their incredible might, could not even touch them.

As the war progressed, the Athenians became more aggressive, sending naval raids to the Peloponnese. Again, the Spartans couldn't really do much against these guerilla tactics, much in the same way the Americans couldn't fight the Viet Cong effectively. I don't know if that analogy helps at all...

Anyways, the Spartans eventually built a navy, and to make a long story short they actually won the war. Athens, though defeated, was able to rise back to power during the Corinthian War, wherein the Spartans basically made the same mistake the Athenians had by being too heavy-handed as leaders of their massive alliance. After that, all of Greece except Sparta was conquered by Phillip of Macedon, and then after that Greece was absorbed into the Roman Republic.

So in brief, Athens was able to compete with Sparta by excelling in fields that the Spartans had no expertise in. The Athenians were innovative, and with the help of democracy they were able to adapt and survive in creative ways. The Spartans, for all their power, could not fight on the seas or use maritime trade in the same way the Athenians could. I hope this answer helps!

I wrote this up before some of the other answers but had trouble submitting so I apologize if it seems redundant.

I'll cover things from the Athenian side in regards to the Peloponnesian War.

Importantly, the Athenians had a far superior navy to anything that the Spartans and their allies could muster. The Athenian navy was famous among the Greeks and its sailors and commanders were very experienced and well regarded. There were many naval battles in the war that the Athenians won despite being outnumbered due to their skill and the lack thereof on the part of their enemies. Even the Corinthians who were considered the best sailors of the Spartans allies could not match the Athenians. In a place like Greece this is very important because so many Greek colonies and cities were on the countless islands in the Aegean Sea. As a result the Athenians were able to control most of these island populations, drawing tribute and support from them throughout the war. The Spartans had difficulty contesting these islands because even if they managed to reach one and install a favorable government they wouldn't be able to support it in the long run.

So why didn't the Spartans simply attack Athens? Because Athens had a very impressive wall at a time when walls were very rare in Greece. In fact, before the war and as a condition of proposed treaties throughout the war the Spartans repeatedly attempted to have the wall dismantled. This wall, which had a segment that extended to the Athenian port at Piraeus, gave the Athenians the option to hole up in the city when Spartan armies invaded Attica. This port access also meant that the Athenians were continuously able to import food from their Black Sea colonies even when their farms were destroyed yearly. And as long as the Athenians stayed behind their wall they took away the advantages the Spartans would have had in open combat.

Commenting from phone so I apologize for any errors.

Sources in advance since I won't be able to reply for awhile:

The Peloponnesian War, Donald Kagan.

The Landmark Thucydides, Victor Hansen.

The answers above are great, but I'd also point you to political science findings as well. "Democratic peace" theory is very well tested throughout human history, with some of the strongest findings. Specifically, democracies don't fight each other, fight autocracies a lot, and they fight with higher intensity and lower casualties, so they tend to win.