They did attempt to do so. They were merely ultimately unsuccessful. From 1638 to 1655 much of the area of New Jersey near Philadelphia was known as "New Sweden" and there were several communities of small hold farmers from Sweden settled in the area. However, the Second Northern War broke out between Sweden and the Polish-Lithuanian Commonwealth the Dutch set up in what is now New York conquered the colony. After the third Anglo-Dutch War in 1673 the territory was turned over to the British.

The mix of Swedish and English claims was a major contributor to the Revolutionary Cause in the area. With the Swedish-descended freeholders being a major group backing the revolution in New Jersey. It's also important to note that the Log Cabin was a Finnish construction method introduced by these immigrants, the C. A. Nothnagle Log House near Swedesboro, New Jersey still exists today and dates from 1638-1643. It includes Finnish Iron Cookware from the 1590's and is among the earliest examples of this kind of construction in the Americas.

But wait, there's more.

In 1650 the Swedes established a colony in the Gulf on Guinea in West Africa known as the "Swedish Gold Coast". They built a number of forts and a couple of factories. In 1656 the Danish conquered the colony with the assistance of the former Swedish colonial Governor. A local uprising expelled the Danes and the Swedes gained control of the colony in the Treaty of Copenhagen in 1660, which ended the Second Northern War. The territory was conquered by the Danes in 1663, and then by the British in 1664.

Sweden also ruled a portion of the Cape Colony in modern day South Africa from 1652 to 1664, before losing it to the organized push by the British to freeze out smaller players in Africa.

St. Barts in the Caribbean was conquered by the British in 1758 during the Seven Years war and was traded to the Swedes in 1784 exchange for trade rights in the city of Gothenburg. It operated as a Freeport and was essential to evading blockades in the region, as much as 20% of US exports were routed through this Swedish colony during the War of 1812. This became much less profitable over time, however, losing Freeport status in 1839, after most of the mainland European colonial holdings in the region had been lost to independence movements. By 1877 it simply was too expensive to keep, so Sweden sold the island back to France.

The Swedes controlled both the islands of Tobago and Guadeloupe for brief periods of time. Guadeloupe was ruled by Sweden for 15 months during the Napoleonic Wars, transferred by British to help solidify an Anglo-Swedish Alliance. Tobago changed hands more than 33 times. Sweden was one of the nations that took a swing at it but wasn't capable of holding on.

Norway, for its part, was under a union under Denmark for much of the colonial period. Despite historically having colonies in Iceland, Greenland, Newfoundland, Orkney Islands, Finmark, and the Sverdrup Islands, almost anything in this period was considered a Danish colony by default.

This was also when the Swedish Empire was at its strongest. The Second (or "Little") Northern War knocked the Polish-Lithuanian Commonwealth out of "Great Power" status. The Great Northern War which started in 1700 and continued until 1720 did much the same to Sweden, resulting in Russia becoming the dominant power in the region. Sweden would ultimately never become as politically, militarily, or economically as important after this conflict.

Sweden was a relatively strong power after their success in Thirty Years War. They controlled a lot of Europe and they even had their own colony in North America, Delaware. They also controlled Finland, most of Baltics, parts of modern day Russia, and even parts of Northern Germany. The Sweden of this era is known as the Swedish Empire. Here's a map that shows them at their peak. It's not the best map I've seen but it's the best I could find online.

The Empire however was defeated in the Great Northern War (1700-1721) which weakened Sweden consideably and they lost most of the lands they had conquered in the thirty years war. They was another conflict soon later, known as Russo-Swedish War that took place between 1741–43, and Sweden was again defeated. Now, you need to remember that Sweden was tiny compared to other European powers. Their population was only around two million and wars like these are very costly manpower wise. A country that small can't hold a large overseas empire like Britain or France could. Sweden also lost Finland to Russia in the early 19th century. Finns had made up around a quarter of the population so it was a heavy blow to the Swedes, especially since they had controlled Finland for 400-700 years at this point.

To put it shortly, Sweden had an empire but they lost it and were too weakened due to Great Northern War and other smaller conflicts to try to create a new one in the 19th or 20th century. There might have also been political reasons but I'm not educated enough on that subject to talk about it.

Norway on the other hand only got it's independence in the 20th century and they were even smaller than Sweden.

Kind of a follow up: Was this due to location, lack of ice free ports, or relatively low population?

The really short answer is that they did. Sweden, and by extension Norway since the countries were in a union for most of the 19th century, had a weak presence in all but Oceania. There was a colonizing attempt in North America, a couple of West Indian islands, Guadeloupe for just over a year and the tiny Saint-Barthélemy for close to a hundred years. Sweden also had a tradepost at the West-African coast and tried to establish one in India.

As to the reasons why none of these ventures, except maybe Saint-Barthélemy and the delve into the slave-trade, was particularly effective it is mostly a matter of size. Sweden was, as it is today, tiny as counted by population and had no economic means to challenge France, Portugal, England or Holland on the open seas. Sweden at this point in time was a relatively poor producer of natural resources as tar, iron and timber. If we add to this huge tracts of arable and usable land inside our own borders that opened up to internal expansion there was some political will but not really the means to go through with colonization.