There's no real consensus on this, so here's an interpretation.

It was planned at a time of unrest and military focus on defence. Remember that it's only one of three walls - Britain, Germany, North Africa - that delineated the edges of the Empire.

Hadrian is reported to have wished to keep "intact the empire," which had been imposed upon him via "divine instruction." This suggests a personal focus on defence and [SPECULATION] a strong desire to maintain and live up to his predecessors' example.

Finally, it's best not to think of the Wall as a barrier but as a form of border control, managing the flow of people and goods.

[Anthony Everitt (2009) Hadrian and the Triumph of Rome, Random House, Inc]

Why do you assume it would cost more? The Roman army was very well practices at building fortifications and the north of England has fairly good stone. Manning it would also not be very difficult: the wall is 73 miles long, so to station a group of, say, twenty men every mile would only require 1460 soldiers. Larger forts could then be positioned to reinforce those if needed. On the other hand, subduing the Brigantes took perhaps thirty years of sporadic campaigning, and wasn't really finished until Agricola marched to the north of Scotland (my theory, from Tacitus, is that the purpose of Agricola's campaign was essentially a psychological show of force). The conquest of southern England was fairly easy because after demonstrating their strength the Romans could fairly easily make the sorts of political arrangements neccesary to pacify the countryside. In northern England and Scotland, however, the political organization of the various groups was too loose for this to work--the Romans could easily defeat their enemies in the field and then force agreements on their leaders, but these leaders would have very little ability to prevent their people from continuing to raid, plunder and otherwise resist Roman rule, particularly if you imagine these leaders, who rule through charismatic authority, to have lost prestige due to their defeat. The Romans could only finally pacify the Brigantes by a combination of beating them into submission and offering significant concessions. This may have been strategically neccesary to protect the southern areas, but no such justification exists for Scotland.

As for the wall itself, I believe there was certainly a very real defensive practicality to it, but it also had a major psychological effect. The Romans were no strangers to using their engineering abilities to frighten their opponents.

It was not for lack of trying, but there were practical difficulties. For one, controlling Scotland required building a lot of forts which required a lot of men, and thus, a lot of supplies, in a place that was, from the Roman perspective, at the end of the world. They kept trying right up to campaigns of Severus, and after that we enter the crisis of the 3rd century and Romans soon retreat back to Hadrian's Wall for good. I know Wikipedia is frowned upon, but the article on this subject http://en.wikipedia.org/wiki/Scotland_during_the_Roman_Empire seems to be of a good standard.

On a side note, conquering tribes living in hills was never easy. For instance, the Romans got the Carthaginian possessions in modern day Spain at the end of the Second Punic War (around 200 bc) and Caesar was still conquering local tribes there in 62 bc, 150 years later. Spain, it bears mentioning, had a lot of silver mines, and it's much easier to supply and reinforce because it's reachable through the Mediterranean sea.

Did they know it was just an isle?

I don't have my books at hand, but Anthony Everitt's "Hadrian and the Triumph of Rome" concluded that Roman forays into Scotland were increasingly costly and fruitless. The land was considered less than optimal and the tribal resistance relentless. (Romans stationed at the Wall almost uniformly despised the posting; it was cold and rainy and muddy and no fun at all).

Pacification of Scotland would require a comprehensive invasion in an inhospitable corner at the edge of the Empire. IOW, it wasn't worth the effort. Instead, continued effective governance of England was profitable and logical. If the Romans could keep what was good and filter what was bad, imperial design and economic value would be maintained. Hadrian understood this and used it as a policy guide during his reign.

Despite the mileposts and hundreds of Roman guards, the Wall wasn't meant to be a physical bulwark against Caledonian raids, even though it was partially effective at doing just that. Instead, the Wall was a border crossing. Much like the Danube, it was filter where civil trade and cultural exchanges could occur. Rather than smash multiple legions head-on into the Caledonian darkness, the Wall would become a static tool of regional pacification.

Let's say you were a Scottish sheep rancher. You collect your wool, life is good. You can now trade your wool to one or more of the warring tribes in your area. One of the tribes is really ornery and won't pay your price, but instead makes threats of violence. The other tribe has recently suffered a harrowing defeat. They've lost land. They're broke. What do you do? Well, you can load up the wagon and head to the Wall. At the Wall, the Roman guards will search you and ask you questions and make you feel uneasy. But in the end, they let you pass and direct you to Pons Aelius, where a Roman bureaucrat will tax you, then vend your wool for a good price.

Repeat this kind of activity hundreds or thousands of times and you start easing border tensions and pacifying the locals. So, despite the monumental cost and effort to erect Hadrian's Wall, it was a smart investment for the Empire.

Did the Romans consider committing genocide against the Northern tribes, instead of building the wall?