In this answer I will start by looking at the history of the Spanish Royal family before the death of Charles

Philip IV was the King of Spain from 1621 to 1665, he had 4 children who survived to adulthood. These were Maria Theresa, Margaret Theresa, Charles II, he also had one bastard son (who was born out of wedlock) John of Austria the Younger.

Maria and John were much older than Margaret and Charles, since Margaret and Charles were born as a result of Philips second marriage to Anna of Austria (also known as Mariana). Anna was Philips' niece, this as well as prior inbreeding in the Habsburg family resulted in their children having very bad health.

In Spain women were allowed to inherit the throne, and since from 1646 to 1657 Philip IV had no living legitimate sons Maria as Philips' eldest daughter was the heir presumptive to the Spanish throne. However when Philip had a son with his second wife Maria stopped being the heir.

Spain had been at war with France from 1635 to 1659. This war came to an end with the Treaty of the Pyrenes, the treaty resulted in Spain ceding a large amount of territory on the Spanish Netherlands to France, the treaty also specified that Maria would marry Louis XIV of France, and that Maria would renounce her claim to the Spanish throne, meaning that her and her children couldn't inherit the Spanish throne. The treaty also required Spain to pay a dowry of 500,000 ecus (gold coins) to France. Spain wanted Maria to renounce her claim to the throne because at this point the relationship between Spain and France was not good, and Spain didn't want to fall under a personal union under France. (A personal union is when two countries have the same king)

This dowry never being paid was one of the pretexts used to argue that the descendants of Maria had a claim to the Spanish throne. Maria had 1 child who survived to adulthood Louis the Grand Dauphin. Louis had three sons of his own Louis the Duke of Burgundy, Philip the Duke of Anjou, and Charles the Duke of Berry.

Phillip the Duke of Anjou(Philip V of Spain) got his claim to the Spanish throne due to being a great-grandson of Philip IV.

Philip IV had died in 1665, and his throne was inherited by his only legitimate son Charles II, since Charles was only 4 years old at the time his mother Anna of Austria ruled as regent.

Margaret married Leopold I Holy Roman Emperor in 1666, who was a member of the Austrian branch of the Habsburg family, unlike her sister Margaret retained her claim to the Spanish throne after her marriage. This was part of an attempt to make sure that the throne would pass to the Austrian side of the family if Charles died without having children of his own. And since Charles seemed to be in a very bad health this was viewed as being a likely event by contemporary observers.

Margaret retaining her claim to the throne was specified in the marriage contract signed in 1663, however Philip IV subsequently was hesitant to allow the marriage to take place, therefore Margarets' marriage didn't take place until after her father died. Anna of Austria who was the sister of Leopold was regent of Spain when the marriage took place.

Margaret had 4 children, her only child who survived infancy was Antonia (more commonly known as Maria Antonia, but I'm not going to call her that in order to avoid confusion). Margaret was regarded as heir to her brother until her death in 1673, after which point her daughter Antonia was regarded as the heir to the Spanish throne until her own death in 1692.

In 1685 Antonia had married Maximmilian II Elector of Bavaria. Maximillian expected that his wife would inherit the Spanish throne, this shows that while descendants of Margaret lived there wasn't much chance for the descendants of Maria to inherit the throne. Antonia gave birth to Joseph Ferdinand before she died, and after her death Joseph was considered to be the heir to the Spanish throne.

While all of this was going on Anne of Austria who was the mother of Charles and Margaret was still ruling as the regent of Spain because Charles II was disabled. Anne was strongly in favour of Joseph succeeding the Spanish throne, and there might have been a plot to overthrow Charles II and replace him with Joseph. Although evidence for this plot is mainly based on rumours, therefore it probably isn't reliable. In any case whether or not there was a plot it ended with the death of Anna in 1696.

Also in 1696 shortly after his mothers death Charles signed a will at the behest of the state council confirming Joseph as his heir. In 1698 Spain once again reaffirmed that Joseph was the sole heir to the Spanish throne, this was done in response to the treaty of the Hague which was an agreement between Britain, France, and the Dutch republic on how Spain would be divided up after Charles died. Other countries making an agreement on how Spain would be divided was highly disliked by Charles II and by the Spanish government.

The death of Joseph in 1699 led to there not being an heir to the Spanish throne, there was a need to fill this vacancy in case other countries attempted to divide Spain after the death of Charles II.

There were two groups of candidates, the children of Leopold I, and the grandson of Maria. After the death of Margaret Leopold had married again and had sons, these sons were Joseph I( a different Joseph), and Charles VI. Since Joseph was heir to the throne of Austria and Spain wanted to avoid being under a personal union, Charles VI was deemed as being preferable to Joseph.

Maria was a member of the Spanish royal family, whereas Leopold wasn't, and if her renunciation of her claim wasn't valid her descendants would be the legitimate heirs to the throne. In addition the regency of Anna of Austria, had led to dislike of Austrians in Spain, therefore the Duke of Anjou(the grandson of Maria) was seen more positively because he wasn't Austrian. The Duke of Anjou was selected above his older brother the dauphin to prevent a personal union.

The treaty of London of 1700 was another attempt to partition Spain, this treaty recognised Charles VI as the heir to most of Spain, while other parts of Spanish territory would be taken by other countries, this treaty was also viewed very negatively in Spain.

After the death of Joseph of Bavaria two factions had developed in Spain, one favouring Charles VI, the other favouring the Duke of Anjou. The faction favouring the Duke of Anjou was led by Luis Manuel. Luis Manuel was a cardinal and had great influence with King Charles II towards the end of his life. Luis as well as others of the French faction convinced Charles II to sign a new will shortly before his death in 1700. The new will left the Spanish throne to Philip Duke of Anjou, if Philip refused the throne it would go to his younger brother, Charles duke of Berry. And it would only go to Charles VI of Austria if both the Duke of Anjou and the Duke of Berry rejected it.

Therefore after the death of Charles II on 1 November 1700, Phillip V inherited the Kingdom of Spain.

Sources:

The Resilience of the Spanish Monarchy 1665-1700 by Christopher Storrs

Heinemann Advanced History: Spain 1474-1700 by Colin Pendrill

The wars of Louis XIV by John Lynn

The War of the Spanish Succession by James Falkner

Europe in the Seventeenth Century by David Maland

War, Diplomacy and the Rise of Savoy, 1690–1720 by Christopher Storrs

The War of Spanish Succession (1701-1714) broke out over precisely this issue, between rival claimants Philip (V) de Bourbon, Duke of Anjou, and Charles (III) von Hapsburg, Archduke of Austria.

Philip of Anjou had the slightly better claim - he was descended from Philip IV of Spain via his grandmother Maria Theresa, who was Philip IV's daughter. Charles was descended from Philip III of Spain via his grandmother Maria Anna, who was Philip III's daughter.

The dispute ended with the Treaty of Rastatt. Charles (who had in 1711 become Holy Roman Emperor Charles VI) renounced his claim to the Spanish throne in return for many territorial concessions by Spain; Philip of Anjou became recognized as Philip V of Spain, in exchange for renouncing his claim and those of his descendants to the throne of France, to prevent a union of France and Spain.