The harem wasn't just a den of the sultan's personal concubines. That's a common misconception. Rather, the harem was simply a restricted area of the palace that was typically reserved for women (and since they needed escorts and guards, said escorts were eunuchs). It's rather likely that the majority of women in the harem weren't concubines at all but rather an assortment of female servants.

The role of the harem was expanded in the later years of the empire. This is most notable from the time of Ahmed III onward. He implemented a change in succession laws. Prior to that, succession to the Ottoman throne was often a fratricidal event involving lots of strangled siblings. Ahmed III decided that the Ottomans should have a more conventional system of succession and essentially started grooming his nephew, the future Mahmud I, for rule. He eventually stepped down in favor of Mahmud, who notably acceded the throne and then didn't immediately murder his brothers. Mahmud I was succeeded by Osman III, his brother. Osman III was succeeded by Mustafa III, the son of the aforementioned Ahmed III. Mustafa III would be succeeded by his brother (Abdul Hamid I) before his son (Selim III) acceded to the throne. You may notice a lot of close familial connections, often in the same generation, and a distinct lack of murder. This is largely because of Ahmed III's reforms. As part of them, the designated successor(s) would be groomed for rule in the harem itself. This marks a rather significant change from the old practice of sending future heirs out to rule various provinces.

The harem itself was basically a small city. It featured a mosque, a tower for resident tutors, baths, residences for the valide sultan (the mother of the sultan) and crown prince (sometimes referred to as "the golden cage"), a reception hall, pool, and an assortment of other buildings. It even had what we might call a pharmacy. It was, in other words, a great deal more than just a place where the sultan selected who would join him in bed that evening.

As for where the consorts came from? They usually came from the Ottoman Empire itself. There are a few examples otherwise (Mustafa III's mother was a Frenchwoman in the harem), but for the most part they were from the Ottoman Empire. Some of them probably wound up there because they were from nobility in the empire (or immediately adjacent to it, like Circassia; Murad V's mother was a Circassian and the daughter of an Ottoman military officer), probably married into the harem (sultans usually had one primary wife and several secondary wives, whose titles would read something like "Third Imperial Consort" in Turkish). Some of the future consorts wound up in the harem because they were enslaved as part of a military campaign; Mustafa II's mother was an ethnic Greek from Crete, which was ruled at the time of the Ottoman invasion by Venice. Some of them have very disputed backgrounds, such as the mother of Cem (son of Mehmet II, he tried to claim the throne, failed, and was shipped off to Rhodes and then Rome.

Afraid I can't answer all your questions, though. But I hope this helped.

as an add-on: how did a harem work in a practical sense? did the sultan just walk in, point to a nearby concubine, and say "you, me, back room, let's go"? did a sultan just swing by his harem on the way to something else, have some quick sex because he had 10 minutes to burn, and then proceed to his original business?

or was it more of a leisurely affair, where he would decide his work for that day was over and just go relax in the harem where he would be tended to by the concubines (not just sex, but be bathed, groomed, etc.)?