Great question!
The reasons for this are twofold: internal and external.
There are some famous stories surrounding the Israeli taking of Jerusalem in the 1967 war, what many Israelis regard as a retaking of Jerusalem. The famous photo of Israeli soldiers at the Western Wall, the holiest site accessible for Jewish prayer, became iconic because of the clear depth of emotion that the moment held for Jews, who had been unable to pray at the Western Wall or Temple Mount while it was under Jordanian rule from 1948-67, and who had faced restrictions prior.
So why would Israel, as you note a majority-Jewish country (and with an even higher majority of Jews back in 1967, at nearly 89% of the population being Jewish in 1965), choose to restrict Jewish access to the Temple Mount and hand it over to the waqf? The moment when Motta Gur, head of the paratroopers brigade that took the Old City, radioed to Uzi Narkiss (the IDF general in charge of Central Command) that "הר הבית בידינו" (Har Habayit be-yadeinu, or "The Temple Mount is ours") was a moment of intense emotion for Israelis. What followed is a moment that has been much reported and much-studied. You see, Gur received a delegation of Arab notables who surrendered the city, according to Michael Oren in Six Days of War, and who surrendered their weapons as well. Gur released them, but had to get directions from local Arabs to reach the Western Wall, since few to no Israelis had accessed the Western Wall in 19 years. Despite sniper fire, the IDF's chief rabbi ran to the Western Wall when he saw it, proclaiming that he had come to that place never to leave it again, and Israeli political leaders, upon hearing the news of the capture, began to designate jurisdictions. The Catholic Church would control their sites, Muslim clerics others, and rabbis would hold Jewish sites. Israel's Prime Minister, Levi Eshkol, planned to head to the Temple Mount, but waited due to the sniper fire and ongoing skirmishes in the Old City. Moshe Dayan, his Defense Minister, ignored it and marched himself to the Temple Mount. He was accompanied by Yitzhak Rabin, future Israeli Prime Minister and IDF Chief of Staff, as well as Narkiss. Dayan suggested to Narkiss the pulling down of a small part of the Old City's walls, a symbol of conquest in ancient times. The IDF's Chief Rabbi, Rabbi Goren, suggested that Narkiss go further, and destroy the Dome of the Rock and Al Aqsa Mosque. Narkiss refused both ideas. Dayan reached the Western Wall, put in a note as is tradition, prayed, and declared that even then, upon Israel's uniting of Jerusalem, the goal was peace. His speech included:
"To our Arab neighbors, Israel stretches out its hand in peace, and the members of other religions may rest assured that all their religious rights and freedoms will be fully protected. We did not come to conquer the holy sites of others or to restrict their religious rights, but to ensure the integrity of the city and to live there with others in brotherhood."
Jews danced at the Wall, celebration and jubilation spread throughout the government.
On the Temple Mount, and this gets to your question, Dayan saw Israeli soldiers had hoisted an Israeli flag over the Dome of the Rock. He ordered it be taken down immediately. He had been viewing the area from Mount Scopus through binoculars, and he radioed Gur, who gave his brigade's communications officer the go-ahead to place the flag up. Dayan radioed Gur and told him, "Do you want to set the Middle East on fire?", and it was taken down. He did the same with the flag raised over the Tomb of the Patriarchs in Hebron. He gave the orders that left the Temple Mount in the waqf's hands, and which removed the prior religious restrictions on Jews praying at the Western Wall.
When Dayan made these decisions, he could have easily been overruled by the Israeli government, though as Defense Minister he had reason to exercise some measure of autonomy in areas taken over by the military, especially during ongoing fighting. But he wasn't. Israeli leaders chose not to tamper with the decision by Dayan to enforce the status quo, agreeing that any such decision to change control of the Temple Mount could lead to a continuation of the fight and the turning of it into a holy war.
So what are the two reasons, external and internal? By now it should be clear that the external reason for Dayan's decision was the threat of continued conflict. Dayan and Israel's government largely agreed that such a move would be so provocative, it would ensure that no ceasefire would be possible, and Israel would lose the world's support and sympathy. Despite Israeli jubilation over the victory, a sense of restraint was felt to be needed to ensure Israel did not overextend. Dayan was at the forefront of that decision, but subsequent Israeli governments ratified the decision. Even so, that doesn't explain the "internal" aspect, which relates to the question of Jewish prayer on the Temple Mount. That particular restriction arises not only out of fear of arousing Muslim ire, but also out of internal pressure from religious Jews to prevent Jews from entering the Temple Mount. Jews are ritually prohibited, in mainstream Jewish thought, from entering the Temple Mount because Jews are ritually unpure and are unable to remedy that. There are a variety of other nuances to this that are important, but generally speaking the point is that there was internal pressure from the Jewish religious authorities to prevent Jewish prayer on the Temple Mount as well, which coalesced with the other motivations at hand. While some prominent rabbis opposed this decision, ultimately the weight of the factors in favor of restricting Jewish prayer, and the internal and external pressures, made it difficult for Israel to ever consider changing this policy.
