As a preliminary, it's important to understand that there was no major foreign security threat to the Tokugawa regime in those days. Unlike their European contemporaries who were constantly intriguing against each other, Japan did not have a legitimate cause to make war. But as you mention, Hideyoshi did invade Korea. After this, two important things happened.

First, it was discovered that a lot of the social prestige which Hideyoshi had attempted to gain by actually invading Korea could be obtained much more easily by simply pretending that Japan's foreign partners were vassals. Hence Korean, Okinawan, and Dutch emissaries were regularly paraded through the country, in a manner similar to the sankin kōtai, the massive and extremely expensive parades which all domain governors were forced to make to and from Edo on a regular basis. These foreign emissaries were treated as if they were vassals making offerings to the shogun.

Everyone involved understood that this was only a matter of ceremony, but at the level of national governance -- that is, among the shogun, the elders, and the governors -- it satisfied any desire that individual leaders might have had to force neighboring countries into submission. As the Tokugawa shoguns shifted the domains away from a state of military preparedness and towards patronage systems to ensure peace and stability, it was only natural that such a foreign relations strategy would develop. There will always be, from time to time, a hothead in government who wants to resolve problems with neighboring countries by force; Hideyoshi was one of these. But wars of invasion are expensive, both in money and in lives, and this was a very happy solution.

Second, Tokugawa Japan did have a frontier into which it expanded, in present-day Hokkaido. As above, the shogun's patronage system had eliminated any consensus for invading neighboring territories, but it also had a very important priority of exterminating Christianity. In the 18th century Russian missionaries reached Kamchatka and started converting Ainu in small numbers. It took many years for news of this to reach the shogun but when it did, it set off all the alarms, and worse, the news arrived simultaneous with an Ainu uprising (which happened for extremely complicated reasons; Brett Walker's The Conquest of Ainu Lands has details). There was a domestic security threat right on Japan's border. So the shogun began expanding into sovereign Ainu territory in Hokkaido. But when it became clear that the Christian threat was not very serious, they actually gave up and retreated in the 1820s, returning Hokkaido to the Ainu!! Then they came back in the 1850s, when Japan was being subjected to gunboat diplomacy and there was a more obvious political imperative to seize Hokkaido before a European Great Power got to it. Even so, Edo intellectuals initially assumed they were going to share power with the Ainu, and Japan did not switch to an explicitly colonial project until the 1870s as they adopted Western (here American) concepts of sovereignty. By that time they had also seized Okinawa.

So basically the Tokugawa lucked out by not having any major security threats until the 1850s, although they did get spooked by Russia around 1800.

In 1592, a 55-year-old Toyotomi Hideyoshi who had unified Japan two years ago wanted to conquer China, something it seems few in Japan at the time wanted. So Hideyoshi sent an invasion army to Korea with an initial army of about 150,000. After initial successes the Japanese were forced to retreat to the southeastern portion of the peninsula, and Hideyoshi tried to gain half of Korea through diplomacy. When that failed, a second invasion was launched with pretty much the same results, and the whole thing ended in failure with nothing to show for it after Hideyoshi's death in 1598. During this period, a political struggle which we don't really know the details of caused the death of Hideyoshi's nephew and adopted heir by seppuku, and Hideyoshi's son was only 5 when Hideyoshi died. The execution of the war and political strife at home also caused disagreement among the men of Hideyoshi's inner circle. This created the condition for Tokugawa Ieyasu to convince many of them to join his side and defeat his enemies to become de facto top of Japan by 1600.

In 1602, a 60-year-old Ieyasu established the Edo Bakufu. There is absolutely no reason to think that another invasion of Korea would be any more successful than Hideyoshi's. He couldn't send a larger invasion force (in fact, if he could send one it'd probably be smaller as few clans would likely want to go a third time), and Japan's navy was no stronger. And there's no indication Korea and China were weaker or China less willing to come to Korea's aid. And Ieyasu has first-hand knowledge on what failure could mean for his clan. Governments were not players in a computer strategy game, and simple existence, not conquest, was more often than not the priority. So Ieyasu set out to ensure that the government he created would exist, by passing laws, ensuring a smooth transition of power, and destroying potential rivals. By the time of the third Shogun Tokugawa Iemitsu, the Bakufu could be said to be relatively stable and on solid ground. By then, however, it no longer has an experienced military as most of the men who had fought in the Sengoku were dead or too old. This was a time when Miyamoto Musashi was asked to supervise one of the clan's participation in quelling the Shimabara rebellion in 1638 because Musashi had participated in gasp two whole wars (1600 and 1614-15).

And, as /u/postal-history points out, Japan had by then already figured out they could get what they wanted (and realistically get) through trade and diplomacy. And yes this is ruling out Japan's actual operations in modern Hokkaido and Okinawa.

Both answers given already are pretty high quality, so instead of focusing on why the Tokugawa didn't expand, I will focus on when they did, for a brief period.

WHile it is true that nothing comparable to the Korean invasion was ever carried out by the Tokugawa, some overseas expansion was carried out. In 1609, the Shimazu clan of Satsuma province launched a successful invasion of the Ryukyu Kingdom, forcing the king to be a vassal of Satsuma. This war was carried out by Satsuma soldiers, but with Tokugawa approval, and Ryukyu, which at this point was considered completely foreign (and was, conveniently, a vassal of China) would remain a Japanese vassal until 1879, when it was annexed as the Okinawa Prefecture.

In fact, this successful operation was followed by other, less successful attempts. Also in 1609, the Arima clan launched a similar expedition to abduct the King of Taiwan and make that a Japanese vassal. This mission failed however, as Taiwan had no central authority, but was inhabited by aboriginal tribes of great linguistic diversity. It managed to abduct some random tribespeople but had little success.

This was followed by another mission to Taiwan in 1616, this time outfitted by an official, the Nagasaki bugyo Murayama Toan. However this was an unmitigated disaster, and only one of 16 ships reached Taiwan where it was lost anyway. In 1627 a Nagasaki merchant, Suetsugu Heizo, tried to convince the Shogun to accept Taiwan as a vassal state by dressing up some tribespeople as an ambassador, in order to circumvent Dutch authority on the island, so he wouldn't have to pay tolls. This was never carried out however, so the Shogun never did it.

Finally, there were also two plans hatched for an invasion of the Philippines, then a Spanish colony, in order to eliminate the flow of Catholic priests to Japan. The first was the brainchild of Matsukura Shigemasa, the daymio of Shimabara, and was called off when he died. The second was planned by two officials who wanted to use Dutch ships, but was cancelled with the outbreak of the Shimabara Revolt in 1637.

In addition to this, some Japanese adventurers like Yamada Nagamasa gained a lot of influence in Siam, but without direct backing by the state.

While this openness traditionally is regarded as having ended in 1639 with the isolation laws, the idea of foreign expansion did pop up later. At various times up to the 1670'es, the Tokugawa considered the possibility of sending troops to help Ming China against the Manchu invaders. In 1675 the SHogun also launched an expedition of exploration to the uninhabited Pacific Bonin Islands, perhaps seeing a potential of expansion. However, they never followed up on this one and the islands would remain uninhabited until 1827.

So on occassion, the SHogunate did consider expanding, but with the excpetion of Ryukyu, it mostly didn't materialize. It is also worth noting that many of these plans were carried out by feudal domains, and not the Shogun. It is important to realize Japan was very decentralized, and that feudal domains kept an influence on foreign policy (often having their own) through the whole Edo period. For the Shogunate, relations to other domains were always paramount to foreign relations, and they were the main source of threat.

A lot has been said, but I'll chime in with a point I didn't see in the top few comments - The Tokugawa Shogunate was primarily focused on controlling foreign contact to maintain internal stability, and expanding outside of Japan would greatly undermine that.

In the early years of the Tokugawa Shogunate, until past the death of Tokugawa Ieyasu for that matter, there was trade being carried out with such locales as Vietnam, Thailand, and the Phillipines, as well as the Ryukyu Kingdom in Okinawa (at this point a vassal state), with the Ainu in the north (many were still living in Honshu at this time), with Korea and with several European powers amongst other contacts. Japanese were living abroad, being hired as mercenaries, working in the service of foreign government as advisors or generals, and generally making inroads in the expansion of overseas trade and contact, including with the Americas.

However, the fear of Jesuit interference in Japanese affairs, bolstered by the presence of Jesuits in the Chinese capital and the colonization and conversion of the Phillipines to catholicism, was a constant issue. The conflict and scheming between protestants and catholics of European origin within Japan was another point of evidence against the allowance of Christianity and Europeans more generally.

The Bakufu had no need to colonize territories (excluding Hokkaido and Okinawa) as it had few threats beyond its borders. There was concern over possible invasion from the new Qing dynasty but that never manifested, and Japan was in a strong enough position to be able to withstand any outside threat while maintaining limited trade with the Dutch, Koreans, Chinese, and Ainu in such a way that left no gray area for Japanese to convert to Christianity or conspire with foreign powers to weaken Japan.

With no necessity of expansion, and every reason to be wary of outside influence on the new system which had ended a century of warfare, they stayed mostly within the borders of the Yamato state established by the beginning of the medieval era with the exception of Hokkaido (Okinawa was made a vassal state but not a settler colony).