It's worth noting that in English, while "doctor" is the common term for "medical professional", the terms "physician" and "medic" are still used, and the three words form a useful three-way distinction:

Doctor: Colloquial

Physician: Formal

Medic: Certain specific contexts, like the military

As for how we got here, Etymonline provides the following:

Meaning "holder of highest degree in university" is first found late 14c.; as is that of "medical professional" (replacing native leech (n.2)), though this was not common till late 16c. The transitional stage is exemplified in Chaucer's Doctor of phesike (Latin physica came to be used extensively in Medieval Latin for medicina).

So coming out of Latin you had a phrase like "doctor of physics" (see the Chaucer quote above) or "doctor of medicine" which displaced the native English word as formal education became more important. Over time this got shortened to simply "doctor" in colloquial English. Why the "doctor" part became more casual and the "physician/medic" part remained formal is hard to say; linguistic changes are often rather arbitrary. It doesn't matter which part of the phrase takes on a formal connotation and which a vulgar one because the distinction itself is the main thing that matters from a communication standpoint.

The entry goes on to discuss similar phenomena in many other languages:

Similar usage of the equivalent of doctor is colloquial in most European languages: Italian dottore, French docteur, German doktor, Lithuanian daktaras, though these are typically not the main word in those languages for a medical healer. For similar evolution, see Sanskrit vaidya- "medical doctor," literally "one versed in science." German Arzt, Dutch arts are from Late Latin archiater, from Greek arkhiatros "chief healer," hence "court physician."