This is a very good question. While I'm not an expert I will share my understanding of the War of Spanish Succession and the situation at the time of the Treaty of Utrecht.

The last Habsburg King of Spain, Charles II, was known from his childhood to be infirm and severely genetically disadvantaged. So throughout his reign all of Spain, and other powers, knew he would be a weak ruler. There were palace intrigues, outright coups, and more importantly multiple open conflicts.

Spain lost Portugal and her colonies (prior to this, they had been in personal union), security in the Low Countries due to the War of Devolution and War of the Reunions against France, Franche-Comté (a huge inconvenience whenever they wanted to send soldiers up the Spanish Road to the Low Countries). In the Pyrenees they had to put up with insurrection in Catalonia. Add to this the ongoing Inquisition, and the bankruptcy of Castille, then the bedrock of Spanish stability both economically and politically.

So basically the perception of the King of Spain was really poor. Charles II married multiple times. First to Marie Louise niece of Louis XIV of France. It was childless, and Marie Louise who by any other measure was considered a lovely young woman died due to depression at age 26. Next he married Maria Anna daughter of the Elector of Palatine. This still did not produce an heir.

Throughout all this, keep in mind the Kingship of Spain could pass through a woman, or descendant of a woman. This means that childless Charles II's crown would be passed through his siblings' lines. Now recall Charles II had two surviving sisters: the eldest Maria Theresa who married Louis XIV, and the second Margaret Theresa who married to the HRE.

His second sister Margaret Theresa's daughter's line had produced a son, and in fact this is what they counted on to be the next King of Spain for years until Joseph Ferdinand died age 6. With his death, they had to go well back up the family tree to find the next legitimate claimant, the two leading candidates being the two parties to the War of Spanish Succession: the French Duke of Anjou and the HRE's Archduke Charles.

More importantly, at that point the entire Spanish Habsburg royal line perished. Thus it was a matter of whether the Austrian Habsburg would take over, or the French Bourbon would win their claim. Both were outsiders, and arguably nearly equally so due to frequent inter-marriages between the royal houses (and the resulting genetic disadvantages). The difference is perhaps that Spain and France had been in war many times. However, this is countered by the rationale that the French had a better claim as it is traced through a more senior line via Maria Theresa.

Yet even before the War of Succession began, there were multiple attempts at "Partitioning" the holdings of the King of Spain. In the first attempt (Treaty of Den Haag 1698), England and France had agreed to recognize the Bavarian claim to Spain, and in exchange Italian lands and low countries were to be divided between them. The rationale was that the Bavarian candidate Joseph Ferdinand would belong to the Wittelsbach dynasty, and thus keep the balance of power between the Austrian Habsburg and French Bourbon. In a twist of diplomatic brilliance, Spain protested the partition by assigning the entire inheritance to Joseph Ferdinand.

Unfortunately Joseph Ferdinand died thus the plan died with him. In the second partition agreement between England and France, the Austrian Habsburg candidate was to be King of Spain, and his Italian territories would go to France as compensation. In neither treaties was Austria part of the discussion (nor Spain), and Austria decided the Italian territories were too important to let go given its wealth and accessibility from Austria. This led to the Spanish court's protest and disgust at Austria's attitude, eventually leading to pro-Bourbon parties coming to a majority. Thus, Charles II finally assigned Duc d'Anjou his successor the entire holding, with the caveat that France and Spain not be united.

Now, looking ahead to the end of the War of Spanish Succession, the Treaty of Utrecht itself isn't just one document, but rather a set of peace treaties between various parties of that war. So just like the attempts at partition, the big powers at the time knew that the lineage of Spanish Habsburg was expiring and thus the big question is really what to do with the holdings around Europe and around the world.

TL;DR Charles II had to choose between immediate family, the Wittelsbach, Austrian Habsburg, French Bourbon. The immediate family choice became non-existent. The Wittelsbach candidacy was derailed by death. The Austrians showed themselves to be more concerned with Italy. Thus in the end he chose the French Bourbon with support of majority in Spanish court.

Edit: added part on the Treaty of Utrecht being a set of small treaties between the powers