Early Rome's borders were a bit fuzzy. They were almost completely void of any fortification, and maintained by complex treaties and alliances with neighboring kingdoms and tribes. Border disputes were common, and often used as casus belli against weaker states. Roman Emperor Hadrian eventually decided to fortify and Empire on a strategic scale, as opposed to fortifying individual settlements. These fortifications were called limites, and manned by special troops called limitanei. Hadrian's Wall in northern Britannia is probably the most famous of these, and is still a British tourist destination today. The limitanei were permanently stationed in an area, contrary to the roman field comitatenses. They were armed with a spear and shield, received very low pay, and sometimes did not even wear shoes.

However, the walls couldn't be expected to hold off any determined attacker, and were mostly used as an expression of power and as a way to separate the lands of "civilization" from the "barbarian" lands outside.

In terms of your second question, the Romans did not have an official customs department, or really even an official police force, only the Cohortes Vigilum, who acted as night watchmen (catching robbers and breaking up the occasional fight while on patrol) and firefighters. Neither the Vigiles nor the Limitanei ever really left their posts, so it is assumed that travel within the Empire was fairly simple, and one could travel pretty much anywhere within without papers. In addition, the limites did not represent the end of the Empire; business was conducted frequently outside the walls, and many citizens traveled through them daily. So, limitanei may have doubled as customs officers for the purpose of taxation. But the first trace of a passport (in the Empire at least) can be traced to the cursus publicus, the messenger system of Rome. The fastest way to get anywhere in the Empire, it used speedier modes of transport than the average person was capable of, and required a diploma, or certificate of permission, from the emperor and the emperor only, to use.

Borders in general were nothing new; city-states like the ones in Mesopotamia at the dawn of agriculture were using large boulders to represent borders, and they built walls as much to keep the borders of the city secure from within as from without.