Unfortunately it's quite a difficult question. They certainly knew that they were potentially playing with fire. The mandate system they designed was intended to deal with the institutional shortcomings of the future nation-states they were trying to create. Key advisers like Gertrude Bell, T.E. Lawrence and others believed that these states would be incapable of immediate self rule. There was quite a lot of recognition about this from Arab peoples themselves as recorded in the private opinions of, for instance, Prince Faisal who was well aware that he needed British military support, and in the recorded opinions of Syrians in the King-Crane Commission who rejected French rule but called on the United States to fulfill an advisory role in the building of the country.

So in that sense, yes, they did know it would cause huge problems.

On the other hand they did not go about this process wantonly. In terms of constraints to keep in mind, particularly in Iraq, is that the borders and constituent pieces of Iraq were not chosen arbitrarily. The solution to the "Mosul question," wherein Mosul province (including Iraqi Kurdistan) was given to the British Mandate of Iraq rather than its original allotment to France in Sykes Picot was designed to balance the Shia/Sunni split about 60-40, thereby making it possible for Feisal to rule, while simultaneously uniting the Arab population (regardless of sect) against Kurdish claims of independence (which began almost immediately with armed rebellion in the mountains essentially from the end World War I) and as a guarantor of continued British presence as it was believed that Feisal could not keep the revolt down without British military advisers and equipment.

And crucially it seemed to work. Iraq was a "model mandate" and was the first of the mandates to receive independence in 1932.

Unfortunately your question would require in depth knowledge of the histories of a couple dozen countries over the course of the late 19th and up to the middle of the 20th centuries, but its worth pointing out as well that much of your question relies on definitions of words like "unstable" that are difficult to quantify. The history of Egypt in the 20th century, for instance, is not a particularly "unstable" one, at least internally. The same could be said of Tunisia, the UAE (formerly the Trucial States) and some others.

There's also the huge issue in your question of assigning blame to European colonial powers for 20th century political developments they had little to do with, of particular importance today being the rise of political Islam in the early 20th century. That is, in and of itself, a profoundly difficult question, as the early history of, say, the Muslim Brotherhood, was integrally linked with anti-Colonialism, but as a group they didn't contribute to the political instability of the country until independence, particularly in the form of street-fighting in the 1930s and an attempt on Nasser's life that led to the movement being banned and its members imprisoned.

TL;DR The question is a complicated one. They certainly knew that creating independent nations in the middle east would be extremely difficult. In countries like Iraq they thought they had come up with a workable solution. Given the range of your question however it's difficult to answer without narrowing it down. If you have any followup questions I'd be happy to try to answer.

Source wise, D.K. Fieldhouse's Western Imperialism in the Middle East 1914-1958 is excellent.