Hey! This is really more a question on how education as a whole worked within the Islamic Golden Age. The House of Wisdom itself, although being famous both in its time and in the modern day, was never a formalized learning institution like a university. It instead was operated in much the same way as education as a whole had been operated for hundreds of years and, after the fall of the House of Wisdom in the Siege of Baghdad in 1258, would continue to operate. The House of Wisdom merely had the added benefit of being a well-funded, centralized hub for learning - similar to how a modern university town may attract a more educated population.

#How Education Worked

When teaching occurred within the House of Wisdom, it was largely under the individual tutelage of learned scholars. This meant that the teaching was an agreement between the teacher and the student, not the result of any sort of institutionalized panel that accepted or rejected candidates. Rather, the teaching within the House of Wisdom would have been part of a more decentralized ḥalaqa system, gradually morphing into a more structured (but still not completely institutionalized) madrasa system.

Ḥalaqa literally means circle or ring and its name denotes how the system works. A Master would sit in a circle with his students and go through various treatises providing commentary. Students could then ask questions, either to the Master himself or to the various advanced students who acted as sorts of Teaching Assistants. This dialogue between teachers and students, rather than mere book learning, ensured that students would prove that they had mastered the material. Although I'm most familiar with the system for teaching Islamic law, the system for mathematics and other sciences would have been similar.

These circles would have existed almost anywhere that there was a local qāḍī or other learned man, typically within the local mosque. However, an ambitious student would not have restricted himself to one circle. He could have attended multiple in the same city or, in many cases, traveled to more famous cities to seek the tutelage of a particularly renowned master. While local circles were often open to all who wanted to attend, they did get more selective as you "moved up" to more and more famous teachers. In these cases, students could bring versions of a "letter of recommendation" from another scholar, asking that the new teacher accept the student. However meritocratic this system may have been, the downside to these circles was that you needed to still pay to support yourself (or attract a private donor to pay for you). This, among other reasons, led to the establishment of madrasas.

#Funding

Around the 8th century, certain mosques began to supply salaries for both their teachers and students. This role then got coopted by Waqfs, or charitable funds, that were set up to supply the funding for these institutions. Throughout the Islamic world, it was common for both men and women to will their money or land after death to the perpetuation of these waqfs, often specifying that the money be used to support education. Gradually, this morphed into the madrasa system of education. What differentiates the ḥalaqa and madrasa system is largely the level of independence enjoyed by the teacher. Wael Hallaq largely criticizes the madrasa for this very reason, saying that it was:

"as much, if not more, a financial and a political phenomenon as it was an educational one, and it subjected legal education to increasingly systematic control by rulers".

The House of Wisdom itself actually fell victim to this sort of control. Caliph Maʾmūn, who really jump-started the House of Wisdom, pumping in funding and emphasizing learning, was a member of the Muʿtazila school of thought. Although now largely defunct, this school were extreme rationalists, with Caliph Maʾmūn going so far as to launch an inquisition, the miḥna, forcing scholars to attest to the fact that the Qurʾan was created and not eternal. Although this repression succeeded during his reign, and partially for the reigns of the following 2 Caliphs, there was a backlash under the Caliph Mutawakkil. Al-Mutawakkil held to a much stricter, more fundamentalist interpretation of religion and openly persecuted scholars who had even just ties to the Muʿtizilis, such as the "Father of Islamic Philosophy", Al-Kindi.

Getting back to funding, in addition to the waqf institutes providing grants for teacher and student salaries and living expenses, scholars could earn generous stipends from local notables for supplying new translations of works, inventions, or astronomical calculations. The Banū Mūsā, who I'll talk about below, reportedly would pay 500 Dinars a month - roughly $18,000 - $20,000 in todays money - to top translators.

The Banū Mūsā themselves, or Sons of Moses, are interesting to note here. These three brothers - Muhammad, Ahmad, and Hassan - had been the sons of the astrologer of the Caliph Maʾmūn. When their father died while they were still young, the Caliph then paid for their own education, giving them access to the finest private tutors in the Caliphate. The brothers later become accomplished engineers, writing The Book of Ingenious Devices which included, among other things, a robotic flute player that operated off variations in air and water pressure. While they would have come from an educated family, seeing as their father was the court astronomer, this example does show how notables could sponsor the educations of those they viewed as worthy.

However, this shouldn't be taken to mean that education was completely meritocratic. It was still much easier for a wealthy family to educate their child than for a poor child to rise up to a position of power (although it could happen).

#Religious Status

Finally, we get to religion. Education was not restricted to Muslims during the time of the House of Wisdom. However, the ability for a religious minority to rise to a position of true power did rely, to a degree, on the relative tolerance of the currently ruling Caliph.

The first example I want to give is of Hunayn al-Ishāq, a Christian translator who, after being introduced to the Caliph Maʾmūn by the Banū Mūsā in 817, reportedly rose to become the director of the entire House of Wisdom by 830. He actually lasted through 9 different Caliphs despite never converting! Other scholars were known to have considered converting for social gain, but it wasn't necessary for all. Certain types of professions actually tended to be dominated by minorities, with Jim Khalili reporting that most of the physicians of 8th and 9th century Baghdad were Jewish and Christian.

But not all Caliphs were as tolerant. The aforementioned Caliph Mutawakkil passed a decree in 850 in which, among other restrictions, he:

forbade their being employed in the government offices or in any official business whereby they might have authority over Muslims. He prohibited their children [from] studying in Muslim schools. Nor was any Muslim permitted to teach them.

This didn't stop converts from being able to prosper however, as evidenced by Ali ibn Sahl, a Jewish convert to Islam who tutored the famous doctor al-Rāzī under the reign of Caliph Mutawakkil. If one didn't want to convert, then there would be significant barriers in their way to achieving education or power.

#Conclusion and TL;DR

All in all, the House of Wisdom would have been most likely funded by the Caliph, especially under Maʾmūn. If a student had proven himself in his local ḥalaqa circles or had caught the eye of an influential patron, he could have traveled to Baghdad to study under one of the Masters and, when there, would have most likely been supported by either that private patron or by a waqf endowment. Finally, while it wasn't strictly necessary to be Muslim under most Caliphs, barriers were occasionally put in place that would have prevented non-converts from rising too high in government.

Well, that got a bit away from me but I hope it wasn't too meandering and that it provides a bit of an answer to your question. If you have any questions please let me know and I'll try to get back to them!

Sources

Pathfinders: The Golden Age of Arab Science by Jim Khalili

An Introduction to Islamic Law by Wael Hallaq

The Decree of Caliph al-Mutawakkil was recorded by al-Tabiri.

Edit: Transliteration