It is generally believed that the Norse people did not displace the Sami people in the south because the Sami did not reach the south in large numbers. The Norse moved in from the south and the Sami from the east/southeast in Finland. It wasn't until the 16th and 17th centuries that the southern parts of the Scandinavian Peninsula were populated enough to move into Sami territory in the north.

Regarding the Norse being indigenous to Scandinavia, it depends. The Norse were first among the "current" peoples to inhanit the southern parts. But in my experience, indigenous people usually refer to a minority people in an area where this minority first lived. The Norse were not a minority in Scandinavia. But you could make the case that they are indigenous if you only consider the "here first" aspect.

Sources: Harrison, Dick. Europa i världen medeltiden.

It is actuelly slightly wrong to look at Scandinavia and ask when the norse arrived there, because Scandinavia was where the norse cultur was developed.

Before I write more I need to remind that most of this knowledge builds on archeology, because we have extremly few written sources from Scandinavia and we get them very late. The first people seems to arrive in Scandinavia in 12000 BC, but we would not call these people norse. They have another religion and doesn't speak norse.

Around the middle of the 3th millinium BC a new cultur often called the battle-axe cultur arrives in Scandinavia. This cultur is very importent, because they bring the indie-european language family to Scandinavia and we assume it is also the cultur that spreads it to most of Europe. They also brings another of changes that is importent when looking at germanic and norse cultur. Patriarchy and focus on chattel husbandry. We are not sure how this cultur spread and people have argued for two main ways. The first is that the innovations in this new cultur where so great that new tribes that came into contact with them simply assimilated it and the other is that it was spread with a people that moved west. Today genetics very much seems to show that it was spread with people from the region north of the Black Sea moving west.

Now let us move to what we call the Nordic Bronze Age, that goes from 1700 - 500 BC, becuase here we starts seeing the proto-germanic cultur evolve. Even though we know very little about the religion in Scandinavia during this time, it seems like we can start recognizing a number of features that will latter be included in the norse religion, such as the sun being pulled over the sky in a wagon and we start seeing pictures of people that might be the gods Thor and Tyr.

The ending of the bronze age is also about the time where we assume that the language in Scandinavia have evolved into a proto-germanic language and the first germanic tribes started to wander south and west from southern Scandinavia. These tribes would come into contacts with the romans around 1th century AD. At this time we can still not really talk about a Norse cultur, because they are still part of a much bigger germanic cultur, but during the next 700 years, during the migration area Scandinavia will become distinctive norse as Scandinavia cultur moves in one direction and the other germanic culturs align themself with christianity. Their language will evolve into norse, they will aquire the runes, Odin will replace Tyr as the most prominent god and a new type of Scaldic poetry will evolve.