They controlled much of Sakhalin since the early 1800's, many surrounding Pacific islands before 1900 including the Kuril Islands, and Korea in 1910. Yet the Potsdam Declaration forced them to give up everything but their home island.
I am not saying that they should have kept them, but I am wondering why there were forced to give up land they seemingly owned for decades without any real protest from the international community prior to WW2. Many of them appear to be legal annexations (as much as they can be) and part of international treaties.
The simple answer is that because they lost a war, and there were no specific international legal norms concerning what territory a state may or may not be forced to relinquish.
The terms of the Potsdam Declaration (and thus the terms of the Japanese surrender) made clear that "Japanese sovereignty shall be limited to the islands of Honshu, Hokkaido, Kyushu, Shikoku, and such minor islands as we determine" (Article 8).
The San Francisco Peace Treaty in 1951 made clear that Japan accepted the confiscation of these territories in Article 2, where Japan renounced all rights, titles and claims.
Essentially Japan was forced to give up territory because the Allied Powers said they would have to, the Japanese accepted this as part of the terms of the surrender, and gave final legal confirmation in 1951.
Well the treaty of San Francisco basically removed Japan's rights to its colonies. We saw a similar thing happened to Germany fallowing WWI. Some islands like the Amami islands were eventually returned to Japan. This was a way for the allies to stop Japan from regaining its power following the war.
Here is the treaty of San Francisco also known as the treaty of peace.