Provided you're building them specifically to stop invaders (as opposed to guarding against rebellion or serving as a local center of administration) you would want to control primarily supply routes, that usually means ports, important roads and rivers. Preferably at choke points like river crossings, mountain passes etc.

Now, the medieval era is a long period of time that saw significant evolution of castle design and function but from roughly speaking the crusades and onwards a castle could be a very serious structure that held a significant garrison. As an example, Krak des Chevaliers could hold a garrison 2000 men strong.

Trying to mount an invasion with 1000 angry men at arms picking at your supply lines and constantly threatening to attack your rear is almost impossible.

As I hinted on earlier castles were usually built for more than one reason and that might affect where they were built. For example, a castle built in or very close to a city might be there as a center of administration, a refuge in times of war, a very in-your-face-reminder of local authority and a deterrent to rebellion. A castle built on a river might serve as a toll station, a control point for supplies and a home for a local lord.

Castles were rarely built exclusively to prevent invasion, castles are bloody expensive things to build and maintain, generally you would try to get the maximum use out of them. For that reason you want a peacetime utility as well as a wartime function. To give you some small idea of how much work building a castle is, the Château de Langeais (the old 10th century keep, not the new castle) is estimated to have taken 83000 working days to finish.

Since the garrison of a castle is mobile the castle might not have needed to be exactly in the right position, you still can't bypass it because the garrison will simply move out and occupy whichever position it wants behind you.