The main reason is that U.S. officials weren't entirely sure who carried out the attack. There was a US investigation of the bombing immediately, but it would be several years before that investigation would come to pinpoint the blame on a couple of Libyans sponsored by Gaddafhi (I think this happened some time in 1993). At first, the U.S. thought it likely that it was the Iranian government that committed the act in retaliation for the accidental shooting down of Iran Flight 655 by the USS Vincennes in 1988.

This lack of clarity about the perpetrator made any type of response, let alone an invasion, impossible. But even if the perpetrator was known, I doubt the Reagan administration would have taken an aggressive response. The last time he landed American troops in the region, the US suffered its worst terrorist attack up to that time with the 1983 bombing of the US Marine barracks in Beirut. Despite the popular perception of toughness, Reagan was very reticent to use military force, particularly against terrorist groups and their state sponsors. David C. Wills, for example, has compiled the data of Reagan's response to every act of anti-American terrorism during his administration, and determined that Reagan responded with military action only .32% of the time, or two times.

There were a couple of other, perhaps less significant reasons, for not invading or responding to Pan Am 103. First, it was near the end of Reagan's term, and he actually thought it best to leave the response to the next administration (similar to what Clinton did in response to the 2000 bombing of the USS Cole). Finally, another reason is that Reagan's counter-terrorism policy had been significantly hampered by the Iran-Contra affair. After that fiasco, Reagan's counter-terrorism policy was essentially to make sure terrorism never made headlines and, if it did, to make sure it didn't stay in the headlines for long. An investigation would do that, whereas an invasion certainly wouldn't.

Sources:

David C Wills, The First War on Terrorism: Counter-terrorism Policy during the Reagan Administration

Timothy Naftali, Blind Spot: The Secret History of American Counterterrorism