This depended on your nationality.

From a perspective of someone traveling to Soviet Union, this was in principle not that much more difficult than traveling to other countries - except that you had to obtain a visa from the Soviet Union and this may not have been as easy to get as visas to most other countries. Those traveling for work could of course get visas (subject to vagaries of the system), but tourism was different. There really wasn't much of foreign tourism in the Soviet Union until fairly late, and most trips that began to be offered in about 1970s-80s were preorganized group trips. (No wandering around outside the officially programmed and "guided" wandering around periods.) I traveled twice to the Soviet Union with my parents, and mostly remember the border guards with AK assault rifles slung on their backs and their humongous fur hats, and the wait at the border while our bus was being searched.

From a Soviet perspective, Lauri Vahtre tells some details about the travails of traveling abroad in his book Absurdin suurvalta: Elämää neuvostojärjestelmässä (Finnish translation of Estonian original, 2010), which, unfortunately, seems not to have been translated to English - a shame, because the book is both eye-opening and darkly hilarious. The following is a summary from pages 98-105.

First of all, it was extremely rare for a Soviet citizen to travel abroad. Soviet Union wanted to minimize the chances for defection and the spread of foreign influences, including eyewitnesses that could undermine the official propaganda where the Soviet system was far superior to the capitalist one. There were basically three ways in which a Soviet citizen could travel: on a work assignment, as a part of a tourist group, or from a personal invitation of someone living abroad. (The latter was mostly an option for people in the Baltic republics, who tended to have more relatives abroad.)

The opportunities were extremely rare and as a result, any chance to travel abroad for work, for a scientific conference for example, tended to result to lots of workplace intrigue, up to and including anonymous denouncements of "competitors" as politically unreliable or otherwise undesirable.

More common was to travel as a part of a tourist group. To get into a tourist group, one had to have recommendations from the director of one's workplace, the party committee of the workplace, the local office of the KGB (most important recommendation), and possibly from somewhere else as well. A list of would-be travelers would then be submitted to "highest authority," either the capital of the Soviet Republic in question or directly to Moscow. When every official had approved the recommendation, one could begin the process to obtain a passport for foreign travel (remember that Soviet citizens generally needed a passport for domestic travel as well, and had to show it even when getting letters from a post office).

To obtain foreign-travel passport, one needed an exit visa from Soviet authorities, and once (and if) that was obtained, one could apply for the for a visa to the country one wanted to travel to. This wasn't always easy, because the officials had to be convinced that one really had a valid reason to travel. According to Vahtre, one Estonian intellectual had wanted to travel to Helsinki just to walk on Esplanadi street, which had been a haunt of many interesting historical and literary figures; he was admonished by the authorities and suspected of trying to ridicule the Soviet government, and he did not travel. It was slightly easier to get permission to travel to people's democratic republics than to capitalist countries, but even there, there were differences: getting permit to Yugoslavia was apparently almost as hard as getting a permit to the West.

But! That wasn't all. Those traveling abroad had to learn a thorough instruction on "how to behave abroad and what dangers to avoid." Travelers were told that the agents of the imperialism would try to cheat and lure the Soviet citizens with all sorts of devious ploys, including but not limited to alcohol, gambling, religion and women. A Soviet citizen caught in such a net would undoubtedly become a propaganda tool; and that was not acceptable, of course.

So travelers had to abide by strict rules, and Soviet tourists were, as a rule, never alone but in groups of three at least. The instructions on foreign dangers emphasized that the capitalist world was far more horrible than it would perhaps look like on the first sight; people were ruthless to each other and problems like prostitution, violence, poverty, disease, unemployment and hunger lurked beneath the shiny exterior. While in every country there were some righteous workers who dearly loved the Soviet Union, they were savagely repressed because of their enlightened political leanings. In short, traveling abroad was a stupid stunt, but in its great mercy the Soviet government nevertheless let its people travel - so that they would become convinced that life in the Soviet Union was infinitely better.

Just in case a Soviet citizen nevertheless became enthralled by the vices of the capitalist system, travel permits were often denied unless the traveler had family members who'd stay in the Soviet Union. Staying abroad was actually a crime that could carry the death penalty or ten years imprisonment, and relatives would suffer as well. Needless to say, one's background had to be squeaky clean, and any hint of anti-Soviet thoughts generally put an end to any aspirations of foreign travel.

Thus informed and with a few tens of dollars worth of foreign currency that a Soviet citizen was allowed to have when traveling (and quite likely with something, like a camera, he planned to sell abroad; there were no official means to exchange roubles to foreign currency), the traveler could actually travel. Almost always, foreign travel was routed through Moscow, no matter if the citizen was living in a republic bordering the country he wanted to travel to. If traveling by plane, the border controls were at the airport; if by train, there were likely to be multiple controls at various points, likely once when boarding the train and at least once more at the border itself.

The border zone itself was divided into a special zone some tens of kilometers from the border, which was no-go zone without a special permit, and the border itself, which was extremely forbidden and where all trespassers would be shot on sight. The border zone was guarded by Interior Ministry border guards, who were largely conscripts. They were usually very thorough and checked just about everything, looking for unauthorized persons when going out from the Soviet Union and anything contraband when coming into Soviet Union.

But the travails of Soviet traveler did not end when crossing the border back to the great socialist motherland. He actually had to write a thorough memorandum, organized by days and hours, about where he had been, what he had seen, who he had met and what he had been talking about. He could also be made to write a story about his voyage for a newspaper or talk about it in radio, and tell - for example - that he saw starving kids scrounging trash bins for food in Paris, lynchings of blacks in New York, people getting beaten by the police in Rome, or how it was totally impossible to obtain a pair of shoes in London.

Experience of this system and how everything was twisted for propaganda purposes is a reason Vahtre gives for being these days very critical about Western organizations that seek to accommodate visitors from totalitarian countries; according to Vahtre, people from those countries visiting the West and returning do not spread the message of democracy and tolerance, but act as believable agents of disinformation. And because travel to West is so greatly prized, they will, according to Vahtre, gladly confirm whatever the system wants them to confirm.