Or from another viewpoint was there ever a time in medicine where not having immediate or frequent access to a doctor improved life expectancy?
Hey there! You pose an interesting question but I think there are some clarifications I have to make before talking. I think a better measure of how living conditions rather than the access to doctors benefits health would be in the form of mortality rate. Life expectancy is very finicky, especially when examining a group's health before the 18th century which had very high infant mortality rates as it will often skew the results and makes the numbers subject to gross misinterpretation.
Now going into your question. During the Middle Ages there were many illnesses that predominately affects the affluent. Due to the wealthy and the nobles rather sedentary lifestyle issues such as: obesity, cardiovascular disease, high blood pressure, cancer, and gout began to arise, and were almost never seen in their poor underlings. Another interesting illness that affected the rich which did not affect the more meek was lead poisoning. Cutlery and pottery was glazed with lead at the time. When these noble and wealthy ate and drank they put more and more into their system which is toxic and eventually will lead to death. Its toxic effects were also much more intense in infants and can cause issues with learning and behavior. Professor Kaare Lund Rasmussen said "In the Middle Ages you could almost not avoid ingesting lead if you were wealthy". This shows that occasionally the lifestyle of the rich and the "treatments" they were receiving did more harm than good; just look at Qin Shi Huang who was given mercury pills in an attempt to make himself immortal but instead killed him.
Something else to remember is that during the Middle Ages there were not really doctors as we know them today. Monks and alchemists were the pseudo doctors of the time as illness was believed to be (especially in areas under the control of the Roman Catholic Church) caused by sinful behavior and their illness as a punishment from God. Due to this lack of understand in what caused the diseases it was very rare a legitimate cure would be found.
Doctors did become slightly more legitimate into the Tudor era, but their treatments were still similar to the Middle Ages, just lacking the religious aspect. These doctors were also extremely expensive meaning the poor were technically receiving better treatment in not being treated at all. Tudor doctors believed illness was caused by imbalance in the 4 humors or fluids in the body: blood, phlegm, yellow bile or black bile. To remedy this treatments like blood letting and inducing vomiting and diarrhea all of which were quite detrimental as they would cause infections which the weakened body could not fight off.
Sadly there were not many innovations in medicine until the 18th century (IMO where it gets interesting). During this time the feudal system has fallen (for the most part) and people who would have been in densely packed areas under rule where infectious disease was rampant now were able to move out to more rural areas, reducing the likelihood of infectious disease epidemics. Innovations in child birth, vaccination and medical teaching all helped more people get medical care all while opiates and laudanum were running rampant in high Victoria society leading to overdose and addictions, which could only be sustained by the wealthy leaving it for the most part inaccessible to the less wealthy.
I hope this answered some of your question although I do hope someone is able to provide some insight to Greek and Roman medicine and society which I know very little about. If anyone notices a mistake, feel free to call it out :)