There are a few things to keep in mind here. First of all, your original question assumes that the image you get of "Ancient Arabia" from Arabian Nights and Disney's Aladdin is an accurate one. However, as Edward Said has exhibited in his landmark work Orientalism, there has historically been many problems with Western representations of "the Orient." One of the tendencies classical Orientalists engaged in was exoticization and fetishization of women from the Orient. One of the paradoxes of Orientalism is that Western observers would portray "the Orient" as conservative and traditional, yet sex-crazed. Therefore, it is useful to keep in mind that such portrayals of women in Arabia are often based off of distorted images by people who tended to hyper-sexualize and fetishize "the Orient" and its women.

But, more to the point, you are correct that modern day Arabia is more socially conservative than it was before, say, the 17th century. It does have to do with Islam, but is not simply due to the "rise of Islam," (which emerged in the 7th century) but rather, the rise of a particular strand of Islam called Wahhabism (which emerged in the 18th century.) In the 18th century, a Muslim scholar from central Arabia named Muhammad ibn Abd al-Wahhab rose to prominence. He had a puritanical and rigid view of Islam and believed that the peoples of Arabia had deviated considerably from what he considered to be true Islam. He advocated a modeling of religious life on the way the Prophet Muhammad and his companions lived. In order to further his strand of Islam, he teamed up with Muhammad bin Saud, a warrior and emir in Central Arabia. Through this alliance, Wahhab and Saud helped establish the first Saudi state in 1744. The first state didn't last long, but the alliance between the Saud family and the religious followers of Wahhab persisted. Finally, in 1932, the Kingdom of Saudi Arabia was declared by Abdul Aziz Ibn Saud, and Wahhabism was its official religious doctrine.

However, this particular strand of Islam is not the universal strand of Islam across space and time as is often portrayed. Indeed, Islamic doctrine during the Ottoman Empire was much less puritanical and strict. The Hejaz region in Western Saudi Arabia (where the holy cities of Mecca and Medina are located) was known as a multicultural and cosmopolitan region before it was taken over by the Saud-Wahhab alliance in 1924. Thus, before the influence of Wahhabism, "Arabian lifestyle" was not as portrayed in Western representations, but was also not as puritanical and strict as Saudi Arabia under Wahhabism. Indeed, Saudi Arabia stands out across the Muslim world for being the most harsh and strict in social attitudes, and that is not due to the rise of Islam, but due to the rise of its own official puritanical version of Islam that is not shared by the vast majority of Muslims.

Side note: it is worth mentioning that what you were asking about was not "Ancient Arabia." Ancient Arabia usually refers to the era that preceded the advent of Islam, i.e. before the 7th century. However, due to the nature of your question it was evident that you were asking why Arabia is so different today than it was earlier, but after the advent of Islam. Thus, "Ancient Arabia" is not accurate as this refers to the era before Islam. Premodern Arabia is a more accurate term for the era you seem to be asking about.

First, you have to know a few things about the Arabian Nights. They are not that Arabian and they were written for a courtly culture in the Abbasid Empire that was heavily Persianate. The Arabian Nights, called simply The Book of 1001 Nights in Arabic, are comprised of various stories taken from an international culture that developed in the Abbasid Caliphate around Baghdad in the 8th-10th century and the stories come from Indian, Persian, Greek, and Arab sources. One of the largest sources was, in fact, a Persian book called Hazar Afsad. It is not necessarily reflective of the broader culture. It reflects the culture of the court which was a mixture of Persian and Hellenistic influences but at a time when the ruling class who produced this literature was Muslim. There was a large contingent of the poetry and literature of the period that celebrated wine, women and song, but think about Hollywood today. The values portrayed in many Hollywood films are considered immoral by many Americans. As I recall from a study I read many years ago the 1001 Nights, there was controversy around them in the Arab World. This courtly life of wine and women was at odds with the scholars' values and that of the Ulema (the religious scholars). There is no evidence that the tales were widely known--no quotations or references amongst scholars, but they were popular enough to have survived in several medieval manuscripts from Egypt, Syria, and Iraq. Also, as an aside, the story of Aladdin and his Lamp are not found in the 1001 Nights. That story was added by the first European translator when he translated the tales into French. He claims to have heard it from a Syrian story teller in Damascus, but no Arab source has ever been found so ironically enough, the most famous "Arabian" story is only known from an 18th century French manuscript.

You talk about the Sultan being wealthy. If you're thinking of Disney's Aladdin, don't. That movie doesn't even follow the story of Aladdin as told in the original tale and that tale isn't even known to be original. The main characters of the frame story are a Persian King (Shah) and his Vizier's daughter Shahrazad. The stories were composed during the Abbassid Caliphate when the Arab rulers were the most powerful empire on the planet ruling a territory that stretched from Morocco to Central Asia so it's natural that this wealth is reflected in the stories.

As to women, this is a topic I'm less well versed on, but I can give you a start. The issue of veiling is complicated. The Quran only instructs that men and women should dress modestly and cover their عورة (Awrah) which is roughly translated as intimate parts and what عورة refers to is disputed. Veiling of the face is not commanded in the Quran and the Hadith (sayings of the Prophet Muhammad) are ambiguous on the subject. It is clear that the Prophet's wives veiled their faces, but this is indicated as being due to their unique status of wives of the Prophet. Some Islamic traditions define عورة as the hair, chest, belly, private areas, arms to the wrist and legs to the ankle. Some say عورة includes the face as well. There is a lot of disagreement. However, then as now, the elite culture of the ruling class often went by different standards and there are stories of scantily dressed dancing girls and the 1001 Nights refer to sexual trysts, alcohol, and all sorts of un-Islamic things. The writers certainly exaggerated for artistic stylistic effects, but at the same time, there are lots of stories in the Nights about humble men and women who abide by the mores of Islam.

There are other tales that might be closer to the "common" people and what they believed and practiced. These include the tales of Juha or Nasruddin Khoja a sort of wise fool who uses his common folk wisdom to outwit thieves and kings. There is also the Sirat epic literature about epic heroes of the Islamic past. Sirat mean path and is often used in the sense of "path of life" i.e. biography or epic. Among the most famous are the Sirat Sayf ibn dhi Yazan, the Sirat Abu Zeid Al Hilali (also known as the Taghribat Bani Hilal), the Sirat al-Zahir Baibars about Sultan Baybars, a Mamluk Sultan of Egypt who defeated the crusaders and became a folk hero in Egypt. These were popular folk literature shared in markets and coffee houses by Hakawati (story tellers) and were much more widely known than the 1001 Nights and much more reflective of the societies that produced them. One of the most popular books was Kalila wa Dimna which is a collection of animal tales (like Aesop's fables) and it is full of simple morality tales retold from Indian originals. Al-Hakawati.net has a nice little collection of Arab folk tales in English.

And finally, remember that these tales are written by the elite for the elite--only the very elite could actually read. These were not tales told around the campfires though they have been based in part on some tales that were. They glorified courtly values for their courtly friends. That culture is gone. You cannot expect the world of the Arabian Nights, even where it reflects the culture accurately, to remain static for 1000 years. We don't have the same values as Shakespeare or Dante and the world they depict no longer exists in the West. Cultures change over time. If you had gone to Egypt 50 years ago, very few women would have worn head scarves as it was seen as "old fashioned." Now head scarves are becoming more and more common.

Arabia is furthermore no longer the center of culture, trade, and empire that it was in the 9th and 10th century. That is a much longer entry but the gist of it is that the old centers of power decayed and trade routes moved elsewhere, especially as Europeans discovered ways around the Arab middlemen by sailing around Africa and off to the Americas. New empires arose in the region (the Ottomans, the Safavids, the dynasty of Muhammad Ali in Egypt, and so on) but they never reached the same heights as did the Abbassids relative to the other powers. Europe outstripped the Muslim world technologically (again, another huge and controversial topic) and were able to dominate the region beginning with the French invasion of Algeria and culminating with the Franco-British redrawing of the borders post World War I giving us, roughly, the borders we have today.