From an earlier answer

The Kaliningrad exclave was very much an atypical example of the Soviet rearranging of national territory and its continued existence as Russian territory after the fall of the USSR reflects its relative uniqueness.

When he was looking to the postwar map, Stalin saw the Soviet acquisition of Königsberg as a strategic priority for the Soviet Union as he thought that the USSR needed an ice-free winter port on the Baltic. In meetings with the Allies, Stalin would claim that this portion of East Prussia was "ancient Slavic soil," and this territory would be naturally be Soviet after the war because of this historical provenance. Although the Western Allies initially agreed to this annexation, the Königsberg issue rose up again at Potsdam when Truman and Churchill floated the idea of an Allied Occupation of East Prussia that would oversee a German administration. Stalin outright refused and boldly asserted that the Soviets would throw out any provisional German administration. Additionally, this Slavic soil belonged to the USSR by virtue of the blood shed liberating it from its German occupiers.

The need for Soviet security also colored how the Soviets parceled out this territory to their new allies and SSRs. While Poland and the Lithuanian SSR received the southern portions of East Prussia, Königsberg became the focal point of a separate oblast. Rather than having it incorporated directly into the Lithuanian SSR, Kaliningrad became a part of the RSFSR. Again, security reasons predominated this as Kaliningrad became the home base of the Soviet Baltic Fleet and Stalin was leery of allowing the newly created Lithuanian SSR access to these military installations. The real collateral damage from the Soviet's preoccupation with security was the oblast's remaining German population, who became seen as a potential fifth column in this military zone. The result was the USSR expelled these Germans in 1947 and brought in Soviet settlers, many of whom were ethnically Russian.

So by 1991, this area had lost most of its original German inhabitants and its population was quite different than its neighboring states. Neither Poland nor Lithuania had any real claim based upon ethnicity to the oblast. In theory, the Germans could have claimed it; despite Stalin's claims about Slavic soil, the region was pretty thoroughly Germanized in 1945 and had been for quite some time. Yet Kohl really did not see the recovery of any lost Eastern territories as a priority in the negotiations over German reunification and he correctly recognized such a position would be counterproductive to the whole project. Part of the 2 + 4 agreement that formalized German reunification included a formal renunciation of any German claims on territory parceled out during the Potsdam Conference.