There were absolutely restrictions, yes. We know about this as a result of numerous first-hand (primary resource) accounts from the period. I'll take one example to lead the way, here. Col. Arthur Fremantle was a British Army officer who in 1863 decided to tour the U.S. Historians aren't entirely sure EXACTLY what Fremantle's specific motivations were (he writes in his book that he wanted to see for himself what was going on, but some suspect that he may have been acting as an informal, non-sanctioned observer for the English government). Whatever his reason, Fremantle went to the U.S. right smack dab in the middle of the U.S. Civil War, and got a remarkable tour. During his trip, he entered the country via Mexico and Texas, which he needed to do to avoid the Union blockade.

This is important, because it demonstrates that not just anybody could stroll into the country's rebellious portions: he effectively snuck in. Anybody looking to gain entry into the Union at this time would have had to go through official channels in the north (going through immigration procedures, such as they were), or surreptitiously via the north or south. This was similar to what was the case overland during the war: if a person wanted to travel between different military districts, but especially from Union to Confederate controlled territory (and vice versa) they would have needed a pass or escort. Spies moved between the lines constantly, so this was a very important precaution for military districts, etc.

During Fremantle's journey through the Confederacy, he was given several passes by commanding officers and generals that saw him from one place to the next, which again demonstrates the need for an escort or proper paperwork to travel around the country at this time. By the time Fremantle made his way to Gen. Longstreet, the battle of Gettysburg was about to flare up, and Fremantle actually managed to sit on the sidelines of one of the biggest, most decisive battles in American history. Afterwards, he was given paperwork attesting to his neutral status as a foreign observer by General Robert E. Lee himself, which actually was enough to get him through to Union lines if you can believe it (Fremantle was subjected to interrogation, but Lee's word on the matter was actually good enough for the Union officers).

One thing that comes up a lot in Fremantle's writing is this need for a pass to get from one place to the other, and frequent remarks amongst officers, both north and south, that he was liable to be shot as a spy without such paperwork. This is hardly limited to Fremantle's writing, as any number of first-hand accounts from the period attest to the fact that soldiers and civilians alike needed a pass of some sort to be able to travel, both within a sides lines, and outside of them.

[Doris Kearns Goodwin, 'Team of Rivals'; Arthur Fremantle, 'Three Months in the Southern States']