It's the most sacred symbol to Muslims. I wonder if during the long Crusades anyone ever attempt to defile it? What was the meaning of Kaaba to outsiders?
My example will relate to Mecca in general rather than the Kaaba, since I am unaware of any other attempts in history at destroying the city.
Reynald de Chatillon was a notorious Crusader who was lord of Oultrejordain from 1175 until 1187. He was a French knight who was held in disdain by both Christians and Muslims, as he had terrorized friendly states such as Cyprus merely for plunder. He was in fact imprisoned by the emir of Damascus for some 15 years, and afterwards he attained his position as ruler of the territories beyond Jordan, giving him proximity to Arabia. Reynald had not given up his old ways, and in 1182 he commenced a series of raids against neutral Muslim vessels on the Red Sea, during which time he made his intentions of targeting Mecca clear. However, Saladin's brother Al-Adil managed to destroy the fleet and Reynald retreated, never again to venture south on campaign.
Perhaps this is not the ideal example, but it is a case of a Crusader attempting to attack the city of Mecca, although his exact intentions with the city are unknown.
The Portuguese made a couple attempts at capturing Mecca after they successfully circumnavigated Africa around the turn of the sixteenth century.
Francisco de Almeida attacked Jedda in 1505 and attempted to conquer Mecca the following year, but was repelled by Mir Husayn al-Kurdi. Affonso de Albuquerque also variously tried to blockade Jedda (a key supply route to Mecca) to starve out its inhabitants and to capture the holy city in 1513 to exchange it with the Mamluks for Jerusalem.
If you'd like to read more, check: Abbas Hamdani, "Ottoman Response to the Discovery of America and the New Route to India," in Journal of the American Oriental Society 101, no. 3 (1981): 326-327.