The question of who was actually engaging in the trading is a pretty interesting and complex one. From one perspective, you can find evidence of a huge variety of different peoples participating in the trade. For example, in the island of Socotra off of Saudi Arabia there are masses of inscriptions on the south Indian Tamil-Brahmi script. Along the desert between the Nile and the Red Sea are inscriptions from various Arabian languages. And of course there is plenty of Greek and Latin. So it is difficult to say who participated given that seemingly everyone did.

That being said, what evidence we have points to that Egypt in particular was the primary origin for most of the traders from the Roman Empire (which makes sense). The one document written by a participant, the Periplus of the Erythraen Sea, was probably written by an Egyptian. There are also a couple inscriptions from Deccan temple dedicatory lists that seem to be from "yavanas" (people from the Roman empire), and they reference Coptos, in Egypt, specifically. There is also a census list from Egypt that mentions one person was not counted because he was in India. From a more quantitative standpoint, during the time period of the Indo-Roman trade a particular type of figurine developed in south India that seems to have been inspired by Egyptian style ones. The most likely explanation is that Roman-Egyptian sailors brought figurines with them that caught the eye of the Indians they met on shore--or perhaps Indian merchants traveling to Egypt took a few home with them. Either way it underscores the importance of Egypt specifically.