The claim that Greece is not well suited for hoplite battle dates to the early 20th century, when Oxford historian G.B. Grundy called this the "paradox" of Greek warfare. I wrote about the question in more detail here.

What Grundy and others suggested to be the sensible alternative was a way of war focused on mountain fortresses and light-armed warfare. They pointed out that most Greek territories were accessible only through a limited number of narrow roads through rugged mountain ranges. It would have been very easy for Greek states to fortify these passes and garrison them with missile troops to block any enemy advance. This would render major invasions effectively impossible, would make urban centres safe from harm, and would reduce warfare to border skirmishes with limited stakes and casualties.

The idea doesn't come out of nowhere. Herodotos makes the Persian general Mardonios mock the Greek way of war as senseless; he wants to persuade Xerxes that the Greeks will definitely come out and fight him in the open, so he pretends that their entire way of war is centred around fighting battles in plains, even though it would obviously be smarter to stick to strong fortifications in the mountains. No doubt that's what inspired Grundy's suggestion: the fact that the Greeks themselves understood the principle. Plato says the Cretans actually do fight this way, since their island is too rugged to allow for any other kind of war. So they fortify the strong places and specialise in archery, rather than fighting as hoplites or horsemen. Finally, Xenophon records several historical instances of Greek states managing to defend their territory by guarding mountain passes with light troops. At least by the early 4th century BC, the strategy Grundy proposed was actually in use.

That said, the terrain limitations of Greece are certainly overblown, and the rise of heavy infantry warfare is not that hard to explain.

Firstly, while it's undeniable that over 80% of the Greek mainland is mountains, this isn't a very meaningful statistic. Most people lived on plains and in valleys; most territories were centred around relatively flat, arable land. Inevitably, most of the warfare between Greek states focused on these areas (and the passes leading from one to the other). In other words, the fact that Greece is mountainous doesn't necessarily mean their warfare was mountain warfare and ought to reflect the challenges of such terrain. Many of the parts of Greece that mattered most were relatively open, and by the Classical period, mostly defended by horsemen.

But secondly, throughout the Archaic and Classical periods, most of the more mountainous regions of the Near East were also known for heavy infantry warfare. The Assyrians' most heavily equipped enemies were found in Urartu; Herodotos describes Lydians and Lykians as heavy infantry; Xenophon's Ten Thousand encountered heavily armoured spearmen in Armenia. While there is no universal correlation between rugged ground and close combat specialists, in this period the pattern is pretty clear. Far from being a region of uniquely heavy-infanry-focused weirdoes, mainland Greece was just another mountainous part of the Eastern Mediterranean/Near East where the way of war had come to be (largely) dominated by spearmen with large shields.

This pattern suggests there is nothing unnatural about people in mountainous areas fighting like hoplites. In fact it suggests the opposite: it seems rugged terrain may have encouraged the development of heavy infantry tactics.

Greek sources happily confirm this. Regardless of the lies Mardonios tells you, the Greeks certainly didn't always march out into level plains to fight. They were well aware of the advantages of strong terrain and fortifications, and often preferred to fight there. Several episodes show the ability of even a few hoplites to hold strong positions (like narrow mountain passes) against any number of opponents. Hoplites naturally gravitated towards hills when looking for a place to encamp, but also to form up for battle; many battles were averted because one side thought the other's position was too strong. They also ran to the hills at the sight of enemy cavalry, which tended to dominate level ground. Even the Macedonian pike phalanx didn't always feel safe in the plains and sought the hills if there were too many enemy horsemen about.

In short, the sources suggested an "ideal" way to fight, involving fortification and missile troops, which the Greeks indeed sometimes used. But generally we shouldn't take the idea too seriously that the Greeks only ever fought as hoplites on level ground (or more ridiculously, that they could not fight anywhere else). They did often fight over plains and valleys, but those types of terrain were the horsemen's domain, and hoplites (like armoured spearmen elsewhere) were more at home in the hills and on the walls of forts and cities.