I believe that you are referring to the "Runamo runes", a set of supposed runic inscriptions located on a granite formation found in a clearing at Runamo [a place name meaning "cliff of runes"), in Skane in southern Sweden. Their existence was noted by Saxo Grammatacus as early as the 12th century, when they were said to be already illegible, but considerable sensation was caused in the 1830s when the Icelandic scholar Finnur Magnusson, the leader of a group of runic scholars who were backed by the very august Royal Danish Academy, announced that he had been able to render a translation, after a year of work, after a sudden flash of inspiration – he called it an "ecstasy" or "trance" – inspired him to read the supposed inscription backwards.

It was then immediately revealed, he said, to be an Old Norse verse inscribed by a force of invaders on their way to fight the decisive Battle of Bravalla (??c.770 CE??). This was a spectacular discovery, not least because the battle – in which an invading Danish army led by Harald Hildetand ["Wartooth", who supposedly lived in the 8th century] was routed by the Swedish King Ring or Hring, best known as the supposed father of the rather better known Ragnarr Loðbrók – had hitherto been supposed to be at best semi-legendary. It was as though an Anglo-Saxon scholar had deciphered an Old English inscription that proved that Arthur's final battle against Mordred at Camlann had actually taken place.

The recovered verse was published to considerable acclaim and lauded as both historically important and of considerable literary merit. Unfortunately for Magnusson, reinvestigation followed, and his discovery was rather quickly undermined by geologists who were able to conclusively show the entire inscription was nothing more than a simulacrum which was the random product of weathering. Magnusson continued to insist on his version of events, even publishing a truly massive defence of the runes in Runamo og Runerne (1841). But his efforts are nowadays referred to, if at all, by scholars of obsession.

Sources

Horace Marryat, One Year in Sweden (1862)

Bob Rickard and Sven Rosen, "The Runamo Runes," Fortean Times 35 (1981)

Well stones with natural markings being confused for runes are a common almost everyday occurrence in Scandinavia and particularly Sweden where runestones are most common. Usually they don't go farther these days than an expert (e.g. the professional runologists at the National Antiquities Board of Sweden) taking one look and saying 'nope'.

I don't quite recognize your story though; even real runestones have been read quite differently when they've been damaged, eroded or written in difficult/archaic language. Dozens of interpretations have been published for the Rök Stone alone, for instance, even thought here's a decent consensus about some parts of the inscription.

So it's not really plausible that you'd have a pseudo-runic inscription where people would agree on what the inscription was and that they were in fact runes of a new unknown kind yet while only having disagreement about the interpretation.

Before the 20th century there were cases where you had the full spectrum with some (correctly) claiming it was a natural phenomenon while others were constructing entire sentences. One of these were the runes that the famous biologist Carl Linneaus visited on July 11, 1776 (a week after the Americans wrote a sternly worded letter or smth) after inquiring with locals about any ancient monuments in the vincinity. Linnaeus failed to read the 'runes' but was impressed by the apparent size of the suspected inscription. Having published about it, they attracted a lot more local attention and a number of interpretation attempts were made, as well as people claiming it to be a natural phenomenon. The issue was put to rest by adventurer Thorild Wulff who visited them in the early 20th century, by which time they'd been forgotten again and were largely covered with moss. And Wullf determined that they definitely were not any form of inscription but small quartz veins in the rock where the quartz had eroded away. (T. Wulff, Linnés ”willrunor” i Göteborgstrakten, in Sveriges natur, 1911) There are a few similar cases.

There are also 'real' runestones, as in 11th century carvings in runestone style, which have fake runes on them. U835 for instance. These are not thought to be some undeciphered writing though; the characters don't really form a consistent neat set, the shapes and the carving in general is low-quality. So the interpretation is those cases are that it's simply an imitation of writing by an illiterate carver.

A good (but overly optimistic) rundown of spurious viking sites in North America is here. The list surprisingly omits the Kensington Runestone of Minnesota (you may have seen it on U-Haul trucks). However, the two which seem most like your point of interest are Dighton Rock and the Narragansett Runestone. These both seem to have had carving done in recent times, so they might not be exactly what you're thinking of.

For what it's worth, there is confirmation of a temporary Norse presence at L'Anse aux Meadows in "Vinland," a.k.a. Newfoundland, right around the year 1000, which matches saga accounts that were later preserved in Iceland. Birgitta Wallace excavated the site and has published on it extensively, and one good report is here. Among other things, the excavators found some butternuts, which don't grow in Newfoundland. Whoever dropped them there must have traveled at least as far south as the St Lawrence estuary. Most likely, it took a season to sail from Greenland to Newfoundland, the Greenlandic Norse stayed there over the winter, then they spent a season exploring further south, stayed in Newfoundland again, and finally returned to Greenland. They left the site in good order and didn't abandon much, suggesting they weren't forced out (despite some saga accounts of hostilities, which might tell us more about what saga audiences wanted to hear than about what actually happened). And there's no indications that women or livestock came with them, so they weren't planning to stay very long. All in all, it looks more like a base camp than a settlement.

The only other confirmed Norse find is the Maine Penny, which was minted in Norway 1067–1093, and it was found at a site occupied by Native Americans 1180-1235. It might be evidence of Native Americans scavenging from abandoned Norse sites, or trade, or even raiding and violence, or it might be a hoax, with someone buying this Viking-Age penny on the open market and dropping it on a known archaeological site from roughly the same period. Among these options, I see scavenging or trade as the simplest and therefore most likely explanations.