They were considered prestigious because they were. They were incredibly difficult to get into -- you could not request to join, you would have to be chosen to get in.

According to the French Regulations of Internal Economy and of Infantry [Section IX, Article 1] issued in 1791:

Grenadiers are supposed to set an example of good conduct and of subordination. They are always to be selected from the soldiers of the most distinguished and approved merit. Every year, on the 9th September, a list of privates to complete the grenadier company is to be formed. Each of the several captains in a battalion will select the 3 most eligible men from his fusilier company to become grenadiers. These selected men must have been serving for at least for 2 years and be at least 173.5 cm [5'7" Imperial] tall. These selected men were assembled, talked about, and examined by the captain, officers, NCOs and two senior troopers of the grenadier company. The captain of grenadier company listens to the reports and remarks made, note down such as appear to him founded and then decides whom of the selected men put on the list to propose to the commander of demi-brigade. The commander of the demi-brigade judging from the reports which have been given to him by the captain will accept only those of the earlier selected by the captain men who "deem worthy of a decided preference."

The Imperial Decree of February 18th 1808 stated in Article 9th:

"The Grenadier Company (...) shall be taken from the totality of the corps, from among the men most appropriate by their height (...) and shall be accepted only if they have 4 years of service and have participated in at least 2 of the following campaigns: Ulm, Austerlitz, Jena or Friedland."

Unlike what /u/SlavicThunder says, Grenadiers in the Napoleonic Wars were not literal grenadiers -- their purpose was not to run up to the enemy and throw a bunch of grenades. The use of literal grenades had died down into the 18th century as line tactics became more sophisticated and flintlock muskets began to take over, creating far more deadly musket formations that did not allow men to just run up, light a fuse, and then sit there and throw it. However, as the need and ability to throw grenades into enemy formations died down the need for elite shock troops who could withstand enemy fire would only grow. One of the major things we see in early 18th century warfare is how indecisive it is -- we see so many battles where neither side can really 'break' the other. Grenadiers were one part of many solution; elite shock troops who could lead an assault and break the enemies back. They were essentially one company out of ~10 who could act as a force multiplier for a battalion and give it some extra 'umpf'.

As said above in the quotes Grenadiers were not something people just went into willy nilly; you would have to be selected by your company captain to be evaluated for the position after you gained significant experience in the fusiliers (regular line infantry) and if you had the height. Grenadiers would be given the infamous bear skin caps (18 inches high traditionally) and special uniforms. They would have thick mustaches, tower over the average man (who would be ~5'2"-5'4" Imperial) even without including the bearskin cap.

So, there you are in a line formation standing across from the French and you're about to clash with their battalion. You have already lost a few dozen men from their voltigeur screening force so you're already pretty damn shaken. Then to top it off you see those infamous bearskin caps and bright uniforms marching toward you. You know who they are; they are 100-140 of the deadliest motherfuckers in that battalion who have spent years surviving and perfecting the art of killing. If only to make it worse you know and will shortly experience that they are taller, smarter, faster, stronger, and more imposing than just about every man in your company. These men could at times have 7 feet of total height charging into men who were on average about 5'4ish.

Really appreciate that for a moment and you'll start to grasp why they have such a prestigious and almost mythological perception. They were truly the best of the best in each battalion and would only even exist in battle hardened battalions. New battalions didn't get grenadier company's until they got the experience to earn them. So when you were in a battle and you saw those bearskin caps charging at you you knew that those were at least 4 year veterans who were the best of the best of those men across from you who were coming for the sole purpose of killing you, someone who may have been in his first battle ever. Yeah.

---

Swords Around a Throne: Napoleon's Grand Armee, John Elting

Well, the idea of 'grenadiers' being elite regiments stems from their original job on the battlefield. As you have guessed, the job was hurling grenades.

Black powder grenades were deadly enough, but not all that reliable, and required the user to be uncomfortably close to the enemy to throw them. This necessitates the recruitment of particularly brave soldiers, who are selected for a large size for the sake of better throwing strength. These men would be well paid and given plenty of motivational aid, such as better uniforms.

In battle, these units would have to move towards enemy fire to a rather dangerously close range, and proceed to throw explosives into the enemy ranks that had a nasty tendency to prematurely detonate. The overhand throw was typically used in these situations, due to the relatively tightly packed nature of the unit, hence the issuing of narrow, but tall and imposing hats , to allow the arm to raise over the head freely, yet look more terrifying.

Over time, the highly dangerous position of a grenadier was lionized, and many regiments today carry the name with pride.