You kind of answered the question yourself. Traveling across the ocean was an expensive and risky proposition, and it was hard to justify making the journey if you're only going to be met with "harsh conditions". For a more contemporary account though, we can look at the Saga of Erik the Red, a Norse epic that comments on why Vinland was abandoned:

[Karlsefni and his company] were now of opinion that though the land might be choice and good, there would be always war and terror overhanging them, from those who dwelt there before them. They made ready, therefore, to move away, with intent to go to their own land.

Also, keep in mind that most people in Europe would never actually hear of the exploits of these Norse explorers, and those who did hear about them didn't necessarily believe they were true. Those who did know about Vinland and believed it existed would still have problems, because they didn't know where exactly Vinland was supposed to be (unfortunately there aren't any GPS coordinates taped to the back of the sagas). It was just too risky to go on a major expedition across the ocean trying to get to a place while knowing only the vaguest hint of where that place might be. Columbus was able to securing funding and sail across the ocean because he at least had a pretty tangible reason for his journey, but even then he was met with a lot of skepticism.