Identities crystallize when groups are put under pressure; it is not surprising that the most succinct and direct answer to your question is found in Herodotos' account of the Persian Wars. The Spartans, worried that the Athenians would give in to Persian threats, sent a messenger to Athens to ask whether they were planning to submit to Xerxes. According to Herodotos, this was the Athenian reply:

We think it is disgraceful to be afraid, especially since we know what Athenians are like: there is no store of gold or territory of surpassing beauty and excellence on Earth that the gift of it should win us over to the Persian side and enslave Greece. For there are many great reasons why we should not do this, even if we wanted to. First and foremost, the burning and destruction of the adornments and temples of our gods, whom we are constrained to avenge to the utmost rather than make pacts with the perpetrator of these things. Next, the same blood and the same language and common shrines and sacrifices, as well as similar customs, all of which it would not befit the Athenians to betray. Know this now, if you knew it not before, that as long as one Athenian is left alive we will make no agreement with Xerxes.

-- Hdt. 8.144.1-3

The passage covers 5 different points on which the Greeks identified themselves as a common collective:

1. Blood (the belief in a common first ancestor of all Hellenes, the eponymous mythical figure Hellen, through whose descendants all Greek kinship groups were ultimately thought to be related)
2. Language (Greek, despite its fragmentation into 4 major dialect groups and many more local dialects)
3. Shrines (the so-called 'panhellenic' or all-Greek sanctuaries, like the temple complexes at Delphi and Olympia, and the festivals held there, such as the Olympic Games)
4. Sacrifices (shared ways to appeal to and please the gods)
5. Customs (a more vague sense of having a similar way of life, probably expressed in matters like dress, social hierarchy, diet, art and leisure, etc.)

In each of these fields, there was of course substantial variation. The myth of common ancestry sat awkwardly alongside the origin myths of each Greek community, most of which claimed descent from migrant groups. Different gods were worshipped in different parts of the Greek world - some unique to specific states - and the same gods were not always worshipped in the same roles. While there were broad ways in which the Greeks (or at least Greek elites) could be said to share a way of life, exemplified by such things as the agora, the symposion and the gymnasion, not all of these things would have been uniquely Greek and not all of them would have taken the same form throughout the Greek world. There are ways in which we now think of the Greeks as culturally unified (such as their architecture or way of war) which actually saw substantial regional variation. As you point out, there was also nothing about the shared Greekness of the Greeks that prevented them from fighting each other at the drop of a hat, and nothing that could encourage them to unite against common enemies (the alliance against the Persians included only 32 Greek states out of a total of many hundreds; there were more Greek states fighting on the Persian side than on the Greek). But group identity does not require actual similarity as long as the members of the group agree on their perceived similarity. The list offered by Herodotos' Athenians expresses the ways in which the Greeks saw themselves as similar, even if it never brought them to act together as one.