(Even though you mentioned Silk Road, I'm explaining it cause its helps me keep track of which aspects linking Rome and Japan I'm talking about)

Indirectly, there was the silk road even during this time, which led goods from Asia into Rome and from Rome into Asia. During the Roman Empire, Japan was more of a tributary nation to China. They were not a "powerhouse" of trade in their own right. Most of the relations between Rome and Asia (major) took place in Sino-Roman encounters (Romans and Chinese, specifically Han dynasty).

There have been roman goods found in Japan, mostly dating to Constantine I. But the question is how did they get there. Was it trade between Chinese and Japanese, where Chinese traded the coins given to them by Romans in exchange for Japanese goods (most likely) or was it Romans trading with Japan.

The key is the Roman coins were found alongside Ottoman coins, leading to a view point that it was most likely just a byproduct of Silk Road Trade going from Rome to Asian Minor to Asian Major and back. Indirect trade and not direct trade. The romans wanted silk, they sent envoys to China to obtain favor. But to actually trade the silk takes going by land from China to Europe, in this day. Meaning multiple countries made up the Silk Road trade routes, rather than just China and Rome. And while Rome may eventually end up with silk, there might have been multiple exchanges of goods between multiple countries which happened before that ultimate trade for silk.

Also at this point India had already started its own sphere of influence based on trade (the reason buddhism was exported to other places in Asia, and one of the few religions spread by trade), and often Roman coins are found in areas where Indian sea trade happened. Japan may have gotten Roman coins through India for all we know. The coins seemed to be older than other coins around it, from that some have taken it to mean a grand scheme of trade led to the coins being in Japan. Also its weird that coins used as trade would just be kept, preserved even. Seems maybe like it might have been a collector. So many possibilities, and we may never know the real answer.

Another example is a glass dish found in Nara provine Japan. It was clearly of Roman origin. But from what we know of glass work at the time, the glass was created and designed in Rome, painted in Persia, and then traded to Asia Major. Again and indirect trade with Japan. This proves that Japan aggressively traded with other countries in the 5th century, but it doesn't prove Rome had knowledge of Japan. The split between East and West Rome had already happened 200 years ago, and by now the Western half was falling. There really wasn't a "Rome" to speak about (this is where I find history weird, the nation is "gone" but the people aren't, and they keep doing the same stuff, so the country is gone but there are people still there, trading, just with a new country name). One of the first "exotic" (foreign) visitors to Japan is recorded in the 8th century and he's from Persia, Rome is gone by then. In the centuries that Rome was a powerhouse, Japan was still ruled by warlike tribes. The North was the Jomon culture, which were hunter-gatherers. Its not until 500 CE or so that the warlike tribes "fall" or combine into one people under Yamato. Around the Three Kingdoms period of China, Chinese started to call Japanese people "Yamato". It wasn't until this time that Japan became more centralized and built a state modeled after the Chinese. They built those cool royal mound tombs. Around 750 CE is the period we know they started to take on Chinese script, Confucianism and Buddhism (cultural trade from India too). This is when Japan started paying tribute to China.

China had its own views on what Japan was they could have imparted to other countries envoys. At this point China saw Japan as a tributary nation made up of pirates. They were literally called "Eastern Barbarians" by the chinese in one of the earliest recorded mentions of Wa, the ancient name for Japan, in Chinese Wō, so Rome may of heard about it, but wouldn't have had a reason to go there.

So as it stands, there is no real verdict out if Rome had knowledge of Japan. We have Silk Road trade, we have trade between India and Rome, we have Sino-Roman relations. We found Roman Coins alongside Ottoman coins in Japan. That's all the information we really have.

Japanese relations with Rome aren't attested (written) on either side. So we really just have to go off of the scraps of evidence we have and piece together a story.