They were tired.

Alexander's army reached the Hyphasis in 326. The army wasn't a static group of people; reinforcements were recruited several times during the campaign, both from the territories being conquered and from Greece. People were also left behind and sent home: all of the Greek soldiers sent by the League of Corinth were released following the death of Darius III (though many chose to stay on), and the cities and garrisons being founded along the way were being populated in part by wounded or old soldiers.

That said, there were still a lot of men in the army who had been with Alexander the entire campaign, since 335. The main source for what happened to cause them to head home (or attempt to, anyway) is Arrian, who relates in unusual and probably embellished detail the debate between Alexander and his men. Although the speeches are probably made-up, Arrian's main source for much of the campaign is Ptolemy's lost history, so this may well have been a meeting that Ptolemy was present for and himself wrote down in significant detail.

Alexander had only the knowledge of Greek geographers to go by, so his attitude was essentially that they were nearly at the actual ends of the earth, around which flowed Ocean, and therefore it would be silly to go home just when they were on the verge of conquering literally the entire world.

Coenus, who was probably a cavalry commander (hipparch) at this point and had been given a lot of responsibility during the Indian campaign, took it upon himself to speak on behalf of the soldiers. His argument was essentially that the men were homesick, their clothes and armour were old, and that it would be better in the long run for the army to return home so that everyone can see how successful they were and how rich they became. That way, Alexander would have no trouble recruiting a new crop of eager Macedonians and Greeks for a second stage of campaigning. It would probably be an overstatement to say that this "convinced" Alexander, but after sulking for a few days, he reluctantly agreed.

So, was Coenus being polite by not mentioning fear as a motivator? Maybe somewhat, but only in the sense that every new conquest involved danger and fear. It's just that the soldiers found the rewards for the first few years to be worth it.

There wasn't necessarily anything ahead that they hadn't seen before, but there was a whole lot more of the same. Plutarch says that they balked at the depth and breadth of the Ganges, and at reports that the forces they would encounter if they went on to fight the Gandaritans and Praesians included six thousand elephants. By that time, Alexander's army was excellent (comparatively) at crossing rivers, and they had fought elephants before. The Persians had them at Gaugamela, and Alexander had just won against Poros at the Hydaspes in part by leveraging the fact that if you kill the mahout, the elephant itself becomes a liability to its own side as much as to its enemy. Since Poros was defeated but lived and became loyal to Alexander, his army even had elephants of their own at their disposal.

But even an army that is well-prepared for the challenges it faces is going to sustain casualties. Despite the actions being "successful" by military standards, a lot of men had just drowned crossing the Akesinos river, and the idea of fighting six thousand elephants is intimidating even if you theoretically know how to do it.

So the refusal to go onwards was probably an accumulation of fear and the kind of existential fatigue that anyone would have after being on the road for nearly a decade, not a response to some single factor that was more terrifying than anything they'd seen before. (What they should have been terrified of, of course, was dying en masse in the Gedrosian desert on the way home, but they didn't know that at the time!)

/u/artemisisdrinkingash has given a decent summary of Arrian's account of the mutiny at the Hyphasis in 326, but it is worth stressing that while Arrian's Anabasis of Alexander is often our 'main' source on the life and career of Alexander, very rarely is he the sole source, and just as rarely an undisputed one. For the main part of Alexander's reign, we in fact have five possible sources to go on, arranged here by rough chronology:

1. Book 17 of the Library of History, written in the mid-1st century BCE by Diodoros of Sicily (Latinised as Diodorus Siculus);
2. Books 11 and 12 of the Philippic Histories, written in the mid-1st century BCE by Gnaeus Pompeius Trogus, but which survives only as a condensed summary by one 'Justinus' (or Justin), probably(?) from the mid-4th century CE;
3. The History of Alexander the Great, probably(?) written in the mid-1st century CE by Quintus Curtius Rufus;
4. The Life of Alexander, written in the early 2nd century CE by Plutarchos (Plutarch); and
5. The Anabasis of Alexander, written in the early 2nd century CE by Arrianos of Nikomedia (Arrian for short).

I have summarised the in-depth issues with each source in this past answer, so I won't recapitulate everything here. What is worth stating is that Arrian's account belongs to what is termed the 'official' source tradition that derives mainly from Macedonian accounts and tends to be more sympathetic to Alexander's perspective, rather than the 'Vulgate' tradition that relies more on Greek accounts that tend towards being more cynical. In addition, we need to understand that the Anabasis of Alexander was not a text that existed in a vacuum, but 'conversed' so to speak with a number of other extant and well-known texts, most prominently the, well, Anabasis.

The Anabasis, written in the early fourth century BCE, is a narrative account of the expedition of the Ten Thousand, a group of Greek mercenaries recruited by Cyrus the Younger in his attempt to seize the Persian throne from his brother Artaxerxes II in 401, only for their employer to be killed in battle at Kounaxa on the banks of the Euphrates. The Ten Thousand then fled north through Mesopotamia and Armenia to reach the Black Sea, from which they then made it back to Greece proper along the coast. Traditionally, the Anabasis has been considered part of the corpus of Xenophon, who was among other things the commander of the Ten Thousand during much of the retreat, but it is worth mentioning that Xenophon himself, in the Hellenika, claims that the definitive account of the Ten Thousand's activities was that of 'Themistogenes of Syracuse', an entirely obscure character who is probably(?) just a pen name Xenophon used, but has very occasionally been invoked to suggest that the Anabasis is indeed misattributed. Whatever the case, Xenophon was an incredibly well-known and widely-read author, and Arrian's choice to name his work the Anabasis of Alexander was not purely cosmetic. Arrian himself, in a sudden digression at Anabasis 1.12, suggests that his aim was to do for Alexander what Homer did for Achilles, and what Xenophon did for the Ten Thousand. And at several points, Arrian quite consciously models his narrative on Xenophon's.

You can see where this is going.

The mutiny at the Hyphasis is heavily played up at the conclusion of Book 5 of the Anabasis of Alexander, occupying Chapters 25-29. To sum it up in brief (using the present tense here to make clear this is Arrian's account and not definitive fact):

• 25: Rumours of the size of the armies of the Indian state(s) beyond the Hyphasis lead the Macedonian army to gather in groups and start openly complaining. Alexander assembles his officers and begins a speech, stating his aim is to either convince his subordinates to go forward or for them to convince him to turn back, and cites the existing scope of his conquests as proof that going beyond the Hyphasis would be no less viable.
• 26: Alexander claims that the army has not far to go before crossing the Ganges and reaching the Eastern Sea, which forms part of the great Ocean surrounding the landmass of the earth; if they turn back, the Indians beyond the Hyphasis and the Scythians around the Caspian would be left unconquered, and pose a threat to the lands that the Macedonians had only just secured; he cites the exploits of Heracles and, indirectly, those of his father Philip, as proof that tenacity would bring results; he also defends his own conduct in having shared in the army's labours and hardships, and notes that the army had shared in the spoils of his campaigns – the officers, in particular, had also been promised satrapies in the former Achaemenid empire.
• 27: Koinos (Coenus) speaks up, claiming not to be attempting to rouse the already-decided army, but rather to attempt to prove to Alexander that turning back is in his own best interests; he notes the attrition suffered by the Macedonians and Greeks in the army and the fact that Alexander had already sent some contingents back before owing to demoralisation and that many in the army had been on campaign for over a decade – even longer if they had served under Philip – and simply wanted to go home; he suggests that Alexander should return to Macedonia, resettle his troops, sort out affairs back in Europe, and return to Asia with a freshly-raised army eager for spoils, calling on Alexander to exercise some moderation and not to burn himself and his army out.
• 28: The other officers applaud Koinos' speech and Alexander adjourns the meeting for the day; the next day he declares his intent to go on with whoever would volunteer to follow him, and returns to his tent to sulk for another two days; the army does not change its tune, and after sacrifices attempting to gain divine sanction for the crossing prove inauspicious, Alexander finally relents.
• 29: Alexander has the army assemble twelve altars to offer sacrifices, holds games, sets about organising his territories in India, and receives an embassy from Abisares, then builds a fleet that will eventually sail down the Hyphasis and explore the Persian Gulf.

It just so happens that the latter part of Book 5 of the Anabasis of Xenophon also involves a series of speeches by Xenophon and others surrounding mutinous rumblings among the troops, although these take up a considerably larger proportion of the book, from the early part of Chapter 6 to the end of Chapter 8. But there are some other remarkable similarities besides, which a short summary of this part should show:

• At this point, the Ten Thousand have already reached the Black Sea near the Greek colony of Trebizond and have since arrived near Sinope, so are in a position to arrange transport back to Greece without threat of Persian attack, unless they were to loiter.
• The first dispute (6.17-34): Xenophon believes it possible that the Ten Thousand might found a city on the Black Sea coast and establish a powerful state in the region, and has Silanos perform sacrifices to assess its feasibility, but Silanos then leaks this to the men, who (basically correctly) believe that Xenophon is trying to prevent them from getting home. After hearing a few brief speeches from the generals, Xenophon concedes that the army ought to return to Greece in one piece rather than be split apart.
• The second dispute (6.37-7.35): Several of the generals come to Xenophon with a new plan, this time to head east and seize the lands of the Phasians in what is now Georgia. Neon, who was not involved with the others, insinuates to the men that Xenophon was going back on his word, leading the troops to gather in groups and threaten mutiny. In his speech to the assembled army, Xenophon mentions that he, if travelling in one ship, would have no ability to coerce a hundred ships to follow him, and that he, as one man, could not overpower ten thousand. He then digresses about an incident at Kerasos which showed the dangers of a mass panic such as what seems to be afflicting the army now. If this continues, the army will either break apart, or become pariahs after committing various acts of impiety. It is resolved that the army will try those who were involved in lawless conduct, and receive purifying rites.
• The third dispute (8.1-26): Accusations of misconduct are levelled against many of the generals, including Xenophon who is accused of assault by some of the men who claimed he had beaten them without prompting. The first man is revealed to have been beaten because he was digging a grave for a sick man who was still alive at the time; no other men come forward after this, but Xenophon pre-emptively covers himself by stating that he hit some men for neglecting discipline or for cowardice, and on occasion to prevent them slacking off, stating it had always been for the men's own good.

As can be seen, there are echoes of Xenophon in Arrian's account that make the overall veracity of the mutiny at the Hyphasis hard to accept at face value. The Hyphasis mutiny's place within the broader scope of the Anabasis of Alexander is an obvious callback to the near-mutiny of the Ten Thousand at Sinope in Xenophon's Anabasis, and several beats in Arrian's account find parallels in Xenophon: the soldiers gathering to complain about their conditions, the overriding concern with returning home, the meetings of officers and their role in persuading their commander towards courses of action, and the idea of a single man having no ability to fundamentally compel thousands. Now, Xenophon makes himself look a hell of a lot better than Arrian makes Alexander, but we must still consider that Arrian's account borrows heavily from Xenophon's and cannot be thought of as a direct conveyance of distilled historical fact from contemporary sources.