Charles V was during his reign, ruler of the largest empire in Europe, why did he ultimately abdicate all his titles and what do we know of his final years?

/u/Aclockworkwalrus I answered a similar question here, where I broke it down to several questions and answers and also here.

Question 1: Why all the inheritances?

It is insightful to go back 2 generations, to the days of the Catholic Monarchs Isabella and Ferdinand. They inherited the crowns of Castille&Leon and Aragon respectively, with a pre-nuptial agreement to rule at practically equal levels. And indeed, they worked together quite well in harmony.

During their joint rule, they pursued alliances with various other powers: Portugal, England, Burgundy, Austria, and to some extent even the French. Alliance with Portugal was key in order to secure peace in the Iberian peninsula, especially with certain anti-Isabella factions having strong Portuguese support. With Burgundy it was because of the trading relations between northern Castile and the Low Countries which were in Burgundian control at the time. The English wanted Spanish support against France as they just concluded the Wars of the Roses. So Isabella and Ferdinand married off their children accordingly:

Eldest daughter Isabella married Alfonso of Portugal, but Alfonso died before they had children. She then was married to Manuel of Portugal who was named heir, but died while giving birth to son Miguel. Miguel was then heir to the thrones of Castile and Portugal until he died aged 2.
Eldest son John married Margaret of Austria, but died young and didn't have issue.
Joanna ("The Mad") married Philip I son of Maximilian I Holy Roman Emperor, who inherited Burgundy from his mother Mary.
Maria was married off to Manuel of Portugal, even though she was originally supposed to marry King James IV of Scotland. Her son John III became King of Portugal.
Catherine married Arthur, heir to throne of England who then died young with no children. She was then married to Henry VIII eventual King of England.

The first two children basically died young and all their issue also died very young, thus somewhat unexpectedly the inheritance of Castile went to Joanna and her husband Philip the Habsburg.

Initially the heir apparent was the oldest son John who had married Margaret of Austria. However, he died soon after the marriage, with no issue. Succession then passed to oldest daughter Isabella and Manuel of Portugal, but neither this Isabella nor her son survived long.

Succession then fell to Joanna and husband Philip, but not without protest from many factions. The Cortes of Castile considered enacting a Salic law but eventually accepted Joanna as heir, but the Cortes of Aragon outright refused to accept her. Further, at this point Joanna's frequent instability meant that Philip would be the de facto ruler.

When Queen Isabella passed away, King Ferdinand tried his best to maintain control over Castile lands however he was rebuffed by the Cortes of Castile who preferred Philip who was considered more pliant.

Eventually an agreement was reached that Ferdinand was to be regent after Philip I died, until such time Charles V would be king of Castile.

During this time Ferdinand married a much younger Germaine of Foix as part of an attempt to sire his own heir that would inherit the Crown of Aragon. Unfortunately this did not come to fruition.

Basically, even before Charles V became King of Spain among other titles, the situation in Spanish lands was already complicated and full of tension, and many factions in Spain viewed him as a foreigner.

When the time finally came that Charles V was to be king, the Crown of Castile passed to him in a relatively straight-forward manner while the established Aragonese swore allegiance to his then still-living mother Queen Joanna instead. Eventually he was elevated to be a joint King of Castile and Aragon with his mother Queen Joanna. Not that the Castilians were meek: as a condition Charles V had to learn to speak Castilian.

After all this, upon Charles' election to Emperor of the HRE in 1519, the Castilian revolted in protest of what they saw as foreign encroachment and expectation of heavy taxes to find Charles' foreign adventures.

So in this long post I've only covered the complex situation of the Spanish side of Charles' empire, and only up to when he finally managed to pacify the populace. Imagine how complicated things are in Austria, and in Italy, and in the Low Countries.

Question 2: Why not give everything to Philip II

On the German side, at the time of his election to HRE in 1519 the young Charles was merely Duke of Burgundy, not yet Archduke of Austria, even though shortly before the election he was King of Spain before it was Spain. That election was extremely hotly contested, with the two leading candidates Francis I of France and Charles.

Charles V was traveling from court to court as was the custom of that time. While was not in German lands, his brother Ferdinand was put in charge of governance in his absence. At this point Ferdinand didn't have much in terms of personal dominion. So in 1521 Charles made him Archduke of Austria to give him more authority in the eyes of his (or their) German subjects.

However, in 1526 he unexpectedly became King of Bohemia and Hungary with the death of his brother-in-law in the Battle of Mohacs, without issue. This means he inherited lands that were contested by, and under attacks from, the Ottomans. This was also the first time a Habsburg was nominally the Holy Roman Emperor while also King of his own personal dominions; previously they were merely Archdukes of Austria and Dukes of Burgundy.

Among other things, Ferdinand was able to be more assertive than he was previously. In 1531 he was elected King of the Romans, which usually leads to HR Emperorship. Further, at the Peace of Ausburg in 1555, he inserted the infamous "Ferdinand's clause" that exempted knights and some cities from religious uniformity. Given all that, it wasn't very hard to imagine that when Charles retired as HR Emperor, Ferdinand took over.

By this point, Charles was likely to be suffering from depression or PTSD, which afflicted the Spanish side of his family, that likely led to his absence from the Diet at Augsburg. As he could not bring himself to consider any of the practical ways forward, he feigned illness. He launched a passive-aggressive campaign against Ferdinand, who acted as his regent and representative, insisting on impossible conditions to be reached. At the same time, there were many arguments regarding the looming issue of succession between him, his brother Ferdinand, his son Philip II, and Ferdinand's son Maximilian II. One sharp point of disagreement was who would inherit the HRE emperorship and how. Ferdinand wanted to keep it in his line of the family, arguing that he had effectively exercised imperial power, and he should succeed Charles as emperor. Thus, his son Maximilian II should be next in line. Charles wanted a sort of time share. Philip wisely went around his own father to negotiate in Ferdinand's favor in order to preserve harmony in the family, after Charles and Ferdinand's sister Mary of Hungary had to be called from the Low Countries to help facilitate negotiations, during which Charles locked himself in a private apartment and refused to speak to his family for days at a time.

TL;DR he had had to fight multiple enemies through his life, putting up with all sorts of different laws and ambitious allies and enemies alike, and he likely suffered from depression.

All of the above, other than the last paragraph which is pure conjecture, are from J. H. Elliott's excellent "Imperial Spain: 1469-1716".