I'm not aware of any direct evidence about whether he left with a group or not, but it seems likely that he did (if you compare it to similar missions like Augustine's it seems likely, his early biographies also have him going with a large following, but they're not considered to have much historical value).

When in Ireland though, he certainly travelled with a group: when talking about his expenses he refers to the presents he gave to kings, and 'the fees I paid to their sons who travel with me' and describes how once people 'laid hands on me and my companions... and everything they found with us they took away'.

If you're interested, the ninth-century Tripartite Life of Patrick gives the following list of his companions (I've added the the job-descriptions from another translation as well when they aren't clear, or different):

These are the four-and-twenty who were in orders with Patrick—viz.,

Sechnall, his bishop;

Mochta, his priest;

Bishop Erc, his brehon [judge/lawyer];

Bishop MacCairthen, his strong man [champion];

Benen, his psalmist;

Caemhan of Cill-Ruada, his youth [chamberlain];

Sinell, from Cill-Daresis, his bell-ringer;

Athgein of Both-Domhnach, his cook;

Cruimther Mescan, from Domhnach-Mescan at Fochan, his brewer;

Cruimther Bescna, from Domhnach-Dala, his mass-priest;

Cruimther Catan and Cruimther Ocan, his two waiters;

Odhran, from Disert-Odhran in Hy-Failghe, his charioteer;

Cruimther Manach, his wood-man;

Rodan, his shepherd [cowherd];

his three smiths, MacCecht, Laeban from Domhnach-Laebhan (who made the Findfaithnech [a bell]), and Fortchern in Rath-Adiné.

Essa and Bite and Tassach were his three artists.

His three embroiderers were Lupait, and Erc, daughter of Daire, and Cruimthiris in Cenn-Gobha.

And this is the number that were in the company of Joseph; and it is the number that is allowed at the table of the King of Cashel, down from the time of Fedhlimidh, son of Crimthann—i.e., the king of the two provinces of Munster, etc.

What we can learn from this about the historical Patrick is very limited (even the text admits that the number is allegorical) but it's interesting, and not entirely useless. The text reflects what a ninth century author considered an appropriate retinue for a great churchman, but, despite christianisation, there was a good deal of continuity in Irish culture, so something like this would have been possible if he was trying to fit in with local customs. A few of the positions are very practical too: the main currency of ancient Ireland was cattle, so someone like the cowherd Rottán would have been necessary for the gifts that the historical Patrick gave out.