I'm not quite sure what you mean here, since the Early Modern Period would in a Swedish context be defined as the period from the Reformation to 1809. But nationalism as we know it is often thought of as phenomenon that arose in the 19th century. (this is of course very simplistic but not entirely untrue)

As far back as recorded history Swedes and Sami (Lapps, 'Finns' in Viking Age usage) have been distinct (as were Swedes and Norwegians). The Roman historian Tacitus in Germania writes about the Suiones and Fenni (probably Swedes and Sami) already around the year 100 CE, which is one of the very oldest written references to Scandinavia that has any recognizable and specific information.

If there ever was a Swedish nation-state (in the true sense of the word) it was before recorded history in Svíþjóð, which was roughly the yellow area, maybe a bit smaller. It was not a centralized state in any sense though, just a name for the country where the Swedes-proper (Svear) lived, who in turn were divided into many more local groupings. The blue area is the Geats (Götar) By the time you get to the time periods of the Saga literature, there's already some kind of political alliance between the Swedes and Geats, with the Swedes seemingly having power over the common king. Then there was also the Gotlanders. (Gautar)

After the Viking Age (i.e. after 1050) the Swede-Geat distinction continues to fade in importance. The term Svíþjóð gives way to Sværike, the kingdom (rike) of the Svear. but at the same time, around 1200 Sweden starts incorporating Finland. Finland was not a state at the time, and like the Swedes had been, the Finnic peoples of the area were split into many groups (Finns-proper, Tavastians, Karelians, Savonians, etc). So Finns as we know them now came about as an amalgamation of those groups that ended up in Sweden, while others (Estonians, Livonians, Veps) weren't. - Much as 'Swedes' in its present day sense is defined by the Norse peoples who ended up under rule of the Svear.

Later in the Middle Ages you have the Hanseatic League arising and Visby (Gotland) and to a lesser extent Stockholm become Hanseatic cites. Sweden also enters a personal union with Denmark-Norway under a Danish king. The Swedish spoken and written in the capital around 1400 is a heavy mixture of Swedish, Danish and Low German. To the extent that Sværike gets its present day name Sverige (the Swedish word 'rike' didn't change - it got swapped for the Danish version 'rige' .. in the country's own name!) So anyway, point is that by the time you have the predecessor of today's Swedish state, it's far from a country belonging to a single ethnic group.

Now, in the 16th century Sweden leaves the Kalmar Union under Gustav Vasa and shakes off both Danish and Hansa influence. Being a skilled propagandist (but with a genuine hatred of the Danes) Vasa strongly emphasized the nationalist aspect of his rebellion against Christian II of Denmark. To the extent that he had his court chaplain Peder Swart plagiarize William Tell stories and substitute in Vasa as the hero travelling the countryside leading his rebellion and evading union loyalists out to get him at every turn. (in later versions this was changed further into straight-up Danish knights; rather than admit the existence of pro-union Swedes)

Now, by this point Sweden had been claiming Lapland and even taxing Lapland, through men known as Birkarlar. Not so much is known about the Sami at this point in time; but it's around this time that many of them are starting to take up reindeer-herding. Which is well-worth underlining; they were both nomadic and non-nomadic groups of Sami, and reindeer herding is both a relatively speaking recent custom and not one that all of them adopted. In 16th century and earlier there are various groups referred to by Swedes and Norwegians by their lifestyle; Forest Lapps, Coast Lapps and Mountain Lapps.

The Birkarl system fades out and in the 1600s Sweden gets to work on building churches, extending the government's reach to the northern parts of the country, and converting the Sami. Which is not to say that some conversion had not started already in the Middle Ages; and is in fact quite evident, for instance in that there is a much stronger veneration of the Virgin Mary in Sami culture, to this day, than the strict Lutheranism of the 17th century permitted. Another aspect of this was expanding the borders; Sweden claiming Sami-inhabited territory all the way up to Barent's Sea which Denmark-Norway also claimed. Hence there was something of a Cold War of church-building to stake claims.

So the persecution of the native Sami religion also began in earnest in this period; although only one for certain and three Sami at most were executed in the 17th century for practicing paganism (as opposed to hundreds of innocent 'witches' among the Swedes, and a half-dozen who'd converted to Catholicism).

From a contemporary Swedish perspective though, this oppression was simply demanding from the Sami much of what was demanded from their own people: Loyalty to the king and country, adherence to the Lutheran faith (not entirely distinct from the former), paying your taxes, registering births/deaths/marriages, and by the end of the 17th century, to learn to read, read the Bible and small catechism. There were some privileges too - Sami paid lower taxes, were not subject to conscription,

Finns were also 'other' but to a lesser extent. There were 'Forest Finns' who'd immigrated to the forests of Dalarna and Värmland. There were 'Tornedalen Finns' who lived up north and spoke their own variant of Finnish (Meänkieli). And there were Finns living in Stockholm; their religious congregation is mentioned from 1533 (and in fact the first sermon in Finnish was held there - not in Finland), the Bible and Catechisms had been translated into that language relatively quickly. It took 100 years until the catechism (as the second book in that language) was printed in Sami. Sweden translated Luther's Small Catechism to all the minority languages of the kingdom in that era (Russian, two dialects of Estonian, Latvian and even Lenape for the Indians living near New Sweden in America). The quality of these translations was often quite poor though.

So who is a Swede in this multi-ethnic kingdom in the 17th and 18th century? That's a much discussed question without any real consensus. The word was not used in a single way anyhow; it depended on context and who you asked. There's a recorded case from the 1680s of a Gutnish (from Gotland, Swedish since 1645) peasant who claimed to be an "honest Dane" and demanded the use of Danish measurements (1 kande = 1.932 liter) for his beer rather than the Swedish standard that'd been created in 1665. (1 kanna = 2.617 liter) Most often the term was used to denote what language one spoke than as a national identity (it seems, other than in cases where it could benefit you legally)

Finns and Sami were still citizens. Both groups had parliamentary representation in the 17th and 18th century. (the population of Swedish possessions, such as Estonia, Livonia and Pomerania did not) Yet they were still clearly culturally 'other'. If you look at 18th century Swedish writings, Scania, Halland, Blekinge, Gotland, Jämtland, Härjedalen and Bohuslän were all being written about as if they and their populations had been part of Sweden since time immemorial, despite having been part of the country for less than a century.

With the rise of nationalism in the 19th century, the idea of 'one people, one country' gained currency. Many of the allowances and exceptions the Sami had had allowing for their traditional lifestyle were curtailed, and with public schooling, the Swedish language was made mandatory for them. However, Sweden was more tolerant than the Norwegianization program, in that the principle was adopted that Sami ought to be Sami ('Lapp skall vara Lapp'), although with a prejudiced and narrow view of Sami as reindeer-herders only. Non-reindeer-herding Sami were expected to learn Swedish and assimilate.

Sources:

Larsson, Lindberg - Nationalism och nationell identitet i 1700-talets Sverige, 2002

Lerbom - För Gud och kung, släkt och vänner - Folkliga föreställningar om svenskhet under tidigt 1600-tal, Historisk Tidsskrift, 133:2, 2013

Lindmark, Sundström - De historiska relationerna mellan Svenska kyrkan och samerna, 2016