First off, I'll clarify what I think of as a castle, in case it's different from what you think of. A castle, to me, is a fortified building or complex built in Europe after the fall of the Western Empire and that had a multifunctional purpose as a residence/fortress/administrative centre. As such, I would not consider a late construction such as this to be a castle, as it was built after the Medieval period and is not a military structure - even though it has the outward appearance of a castle and would be called a castle by most people.

Castle architecture was typically much more simple in real life than it was in fantasy books.

Consider, for example, Beaumaris Castle, one of the largest and most complex on the Island of Britain. It was an enormous drain on royal finances, and was never completely finished due to its immense cost. Most castles were smaller and simpler than this.

Look at the floor plan of Beaumaris.

Where would the secret passages go? From one of the two rooms in the main towers to the other? It would be kind of pointless.

Most castles, however, were nowhere near the scale of Beaumaris.

For an example at the other end of the spectrum, let's consider Little Cumbrae Castle, in Scotland. It is fairly typical of fortified residences in Scotland in the late Medieval era. It likely had a ditch and wall around it when it was in use (and so likely would have been a castle in the parlance of the times - castellum typically indicated a fortress with an enclosed bailey, as opposed to a single building with no enclosing wall).

Look at the floorplan and elevation.

Again, where would the secret passage be? With the exception of the ground floor storage area, each floor is pretty much one big room.

So, the TL;DR would be that no - castles did not typically have elaborate secret passages of the sort seen in fantasy novels. Their design was typically very simple and straightforward, very practical. These are military structures.

It depends on what you mean by secret passageways. It was very common indeed for castles to feature one or more postern gates. These were small, highly defensible reinforced doors, basically, set in an out-of-the-way or difficult to attack location, and used for slipping out messengers or counterattacking sallies. If we look at Chateau Gaillard, for instance, we see a postern gate at the end of the castle overlooking the river, an area largely invulnerable to attack but ideal for slipping people out, especially at night.

Part of this legendry may be based on the presence of priest-holes in late medieval/early modern English houses. When England was converting to Protestantism (and back to Catholicism, and back to Protestantism, etc.) in the 16th and 17th centuries, it was relatively common for well-off crypto-Catholics to hide priests in their houses. If the houses were searched, they would want the priests to be hidden, of course. So they would build priest-holes, which are secret rooms/compartments for the priests to hide in. They don't necessarily go anywhere, as the priest just needed to sit tight, but they're similar enough to secret passages that they could definitely have been conflated in the popular imagination.

There is this. I'm not sure if this qualifies as a secret passage, though.