There were a few ways of protecting borders, though it is worth noting that borders were often more fluid or more "theoretical" than today.

• Set borders on rivers or other natural barriers This is fairly typical especially in cases like the Carolingian Empire where they effectively got to "pick" where the empire ended and other areas began. Having a natural boundary marker both helped conceptually and provided natural protection.

• Build fortifications. This could take the form of fortresses or castles (depending on the period anything from motte and bailey to stone castles) or of walls, such as the Danvirke or Offa's Dyke, both of which served as much to conceptually identify where borders lay as to actually defend. These would not have been guarded in the same manner as say Hadrian's Wall at its height but they would inhibit raiders (especially if those raiders were trying to move cows...)

• Guard/Fortify natural crossing paths: People tend to move along common paths, except in specific circumstances. Trade travels along convenient paths or roads and thus crossing points could be fortified and guarded. We know that royal (or ducal etc.) officials collected tolls and kept watch on these crossing points. In the Carolingian Empire, for instance, it was forbidden to trade certain goods (blades and armor especially) with the Avars and so we know that border officials would be checking for this sort of thing as well.

As to whether there were passports, we do have references to documents of this sort, in the Lombard kingdom for instance, though its not clear how frequently these would have been used and it isn't my specialty. Diplomats and traders might also have letters of introduction that could aid in free movement and warm reception, either from their lieges (in the first case) or from friends to other friends (in the latter).

Edit: it should be noted this is fairly generalized and focused in the early medieval period, certainly things like border fortifications and set crossing points remained prevalent but I'm sure that things got "more complicated" in the later Middle Ages.