They weren't marching off into an unknown land in the middle of nowhere - they were going to the heart of an extremely ancient and urbanized world, which had long been part of the Roman Empire in previous centuries and numerous other empires for thousands of years before that. Medieval people often went on pilgrimages to Jerusalem by land and by sea. For the most part, they knew exactly where Jerusalem was!

The First Crusade never really went outside the territory of the old Roman Empire and there were still plenty of Roman roads around. The crusaders who sailed across the Adriatic, for example, used one of the main highways of the Roman/Byzantine Empire, the Via Egnatia, to get from Dyrrhachium to Constantinople. Other groups of crusaders followed the roads along the Rhine from France and Germany down to the Danube, and then to roads through Hungary and Serbia passing south to Constantinople - all extremely ancient and well known routes, through very populous areas.

Anatolia had been part of the Roman Empire too and it was also full of ancient roads. The crusaders brought Byzantine guides with them from Constantinople, including a Byzantine general named Tatikios who remained with with them for part of the journey. There were hostile Seljuk Turks in Anatolia but there was still also a huge Greek and Armenian population who were friendly to the crusaders, and who gave them supplies and showed them where to go.

Moving south from Antioch, in addition to the Roman roads, there were even more ancient roads dating back thousands of years. The "Via Maris" is the north-south road along the Mediterranean coast. The crusaders had no problem following it south. They were in Muslim territory then, but there were still plenty of friendly Christian inhabitants willing to help if they needed directions.

This was also not a journey that had never been done before. Some crusaders had been to Constantinople before, and some of them had even gone all the way to Jerusalem as pilgrims. A few decades before the crusade there was an enormous pilgrimage to Jerusalem from Germany, the "Great German Pilgrimage" in 1065.

Of course there probably were some individual crusaders who had no idea where they were going, especially in the first wave early in 1096, the "peoples' crusade". Supposedly some of them exclaimed "ah, Jerusalem!" whenever they saw a city, which they weren't used to seeing back in France and Germany - whether it was Budapest, or Belgrade, or especially the biggest city of them all, Constantinople. But the sources for the "peoples' crusade" are usually written by the people who participated in the second, better-organized wave of the crusade and they were pretty hostile to the first wave. They might just be joking.

Certainly most people knew where Jerusalem was because it was always connected to the wider world through very old roads.

Sources:

Thomas Asbridge, The First Crusade: A New History (Oxford University Press, 2004)

Jonathan Riley-Smith, The First Crusaders, 1095-1131 (Cambridge University Press, 1997)