Christianity, as far as we know. In Erik's Saga, Chapter 5, the recently baptized King of Norway, Olaf Tryggvason, commands Leif Erikson to convert the people of Greenland, which he does after discovering Vinland and rescuing some shipwrecked sailors. His father Erik the Red "was reluctant to abandon his old religion; but his wife, Thjodhild, converted at once." According to legend she built the first church in the Americas, a modest wooden chapel in Brattahild, today known as Thjodhild's Church

Suffice to say, the two Vinland sagas contradict one another at certain key points, and in the Saga of the Greenlanders events play out very differently. Leif does not meet with King Olaf, nor does he convert the people of Greenland. In fact there's a line in Chapter 2, "Greenland was still a heathen country at this time." That said, when Thorvald Erikson explores Vinland in Chapter 5, and gets himself mortally wounded in a scrap with some Native Americans, he commands his men to place two crosses over his grave, one over his head and one over his feet.

So Leif Erikson and his men were probably Christians, but since the sagas were written by some Christ-biased monks more than a century later, who really knows? Personally I like the idea of hard-drinking Odin-worshippers discovering America.

TL;DR - Probably Christianity but there's room for doubt

Edit: got rid of a typo

Hi. This is pretty hard to answer due to the lack of recorded history on the subject. The only indicator is from  The "Saga of the Greenlanders" and "The Saga of Erik the Red". In addition there are a few short sentences in theIcelandic Annals, a couple of geographic treatises, a paragraph in a work by an eleventh-century German priest, and a few maps, that's pretty much the whole source on the whole Vinland situation.

Anyway - religion is mentioned in one of these documents but it is a side note, described in the Vinland sagas, Gudrid becomes an unwilling but effective participant in a seidr ceremony.

So that would lead us to believe it was common norse mythology In place. Seidr ceremonies tended to be more "magic". So chanting and getting wasted is the common theme but there's an abundance of info online about viking ceremonies, you'd want to concentrate on the odin flavoured ones as he was the diety most linked with seidr/magic whereas, for example, Iceland would tend to worship Thor in a different themed ceremony. (But they still got wasted!) Hth

Edit: comment below rightly points out the sources were written by Christians so "unwilling", in my view, probably means willing but like I mention the info is sketchy. But I'd rather think of them as getting smashed and cheering odin in the USA!

Strictly speaking, there probably never was a settlement of Norse men in North America outside Greenland. Their activities were mainly exploration, rather than settlement.

The settlement of Greenland and exploration on North America coincide with a period where christianity is becoming the official religion in Norway and Iceland, and probably Greenland to. But conversion probably took a lot longer time than indicated in younger written sources. A norse sailor who is officially christian for a few months would hardly have a great knowlegde of christian canon or rituals, and quite likely to hold on much of his pagan folk lore.