The Praetorian Guard had a different mission from regular Legionnaires but it's not really analogous to what we usually call "special forces" today, but more along the lines of a gendarmerie.

"Special forces" encompass units with many different missions today. I think it would be more useful to ask about the establishment of units for specific missions - for example, the Italian Navy established a unit for combat frogmen in 1939 (1a Flottiglia Mezzi d'Assalto, later reformed as the infamous 10a MAS). The Nazis formed a special unit of Polish-speaking soldiers to operate in civilian clothing for special missions for the invasion of Poland, the Ebbinghausers, later reformed into the Brandenburgers.

Things get a bit more complicated for certain missions that were "special" back in the day but have become part of the repertoire of regular forces - for example, paratroopers were certainly "special" in their origins but I don't think anyone considers the 101st Airborne "special forces" now. Conversely, a lot of "special" missions have been performed by regular units, and ad-hoc units formed for specific missions. I don't imagine you're asking about these, though.

More generally, specialized units of assault engineers and reconnaissance troops (missions which today often fall to "special forces" even if they're more or less regular) have been around for about as long as there's been an organized military.

Special forces are pretty tricky concept. The definition you are using (troops specialised and trained in "unconventional warfare") presupposes the existence of a "conventional" warfare, which focuses on regular forces engaging in pitched battle in the open. Neither this form of conventional warfare nor the concept of troops specialising in alternative types of fighting translates very easily to the ancient world.

Take Classical Greece for example. A lot of people claim that pitched battle between heavy infantry phalanxes was the "conventional" form of warfare. If we accept this, then all troops that did not fight as heavy infantry in pitched battle would count as "special forces" - which, as you'll probably guess, meant that every army and every engagement involved "special forces". If we therefore decide to define conventional warfare more broadly, to include sieges, skirmishes, raids, ravaging and positional warfare, then there is suddenly nothing but conventional warfare, and no "special forces" at all.

This goes to show that "special forces" is not only a modern concept, but one that builds on several other modern concepts, without which it cannot exist. No Ancient Greek city state had a "military" in the modern sense. They barely had regular training, let alone specialist training. They had no military academy or officer class, and fought their wars in pretty much whatever way they could. In such a world, what would "special forces" look like? How would we be able to distinguish them from the varied temporary units and formations that made up the average militia army?

Probably the closest thing to "special forces" in Classical Greece were the ad-hoc units known as logades or epilektoi (both names mean "picked men"). We see these popping up in a number of Greek armies (the Athenians, the Eleians, the Syracusans, the Ten Thousand) when a small unit was needed to complete a special mission. They were used as combat reserves, assault companies, quick response units, vanguards, rearguards etc. However, they were picked from the regular militia only at need, and received no special training or equipment. Who was chosen depended mostly on physical fitness and loyalty to the cause.

A couple of these picked units were eventually established as standing forces - most famously the Theban Sacred Band. However, once they were formally organised like this, they tended to become more like the elite "shock troops" that you've defined as different from "special forces". You'll have to tell me if the ad-hoc logades count.

We see dedicated combat engineers from the 18th century, but the Foreign Legion Pioneers have been a unit since 1831.

Roger's Rangers in the French and Indian War and Knowlton's Rangers in the American Revolution stand out.

There have been a lot of different special tactics units in the last 250 years in the world. It would be hard to point at one and claim it's the first.

The problem really lies with the actual definition of "special forces" (or more correctly specialized forces) as opposed to the public perception about them. Military units with specialized skills for specific battlefield roles have existed since antiquity. However, I rather doubt that most modern readers would equate a Napoleon-era French Voltigeur (skirmisher) with a modern SEAL team member.

Really, for the modern reader, the term "special forces" in fact often translated to what the British would term as "Commandos" in the Second World War - meaning they are small, independent units capable of operating in many different kinds of terrain, often alone in what is considered "enemy territory". They are distinct from guerillas or partisans - as guerillas/partisans in fact live in the territory where they operate in. Commandos by contrast are not native to the land.

Back in World War 2 these units were often at least company-sized (100+ men) but SEAL team and Hollywood mythology has combined to shrink the size down to a squad. Rambo and Metal Gear Solid have attempted to shrink that down even further to just one man.

In any case in reality they are actually very rarely employed in the original "Commando" role of striking alone in enemy territory. Instead it is much more common for them to be used as line infantry - supporting regular units in regular infantry tasks. Indeed, I would argue that the Rangers - who have a tendency to insist that they are "special forces" - are in fact just line infantry with more stringent entry requirements.

As for the oldest example... I would probably count the advance / reconnaissance parties of various armies of antiquity as technically performing these same roles; albeit some may argue that having a regular army behind them in support makes them less macho than the Commandos. In which case the cavalry raids of the Civil War - often taking weeks at a time and without an army in support - would be the earliest and closest examples.

Edit: Noted the difference between guerillas and commandos.