The answer here is pretty simple -- the United States was still a medical backwater, and Philadelphia, with its University of Pennsylvania, had, in this period, the finest medical school on the continent. Medical schools in the United States were, lets just say, heterogeneous. I've written about this before about the history of my alma mater, Tulane, founded as the Medical College of Louisiana in 1834. And while there were a couple dozen medical schools on the continent at the time, most were similar to Tulane -- sit for some classes, pay the fees, essentially do an apprenticeship (or not), and get a degree. Even Harvard was largely a diploma mill. McGill and the University of Toronto, up north, stuck to a more English university model, but at this point they were small institutions and decades from their heyday in the late 19th C.
And then there was Penn. Penn was well-endowed and supported by the elite of Philadelphia from the beginning, and was explicitly modeled on the British university system (especially the University of Edinburgh). It was also associated with the greatest physicians of the new United States, especially Benjamin Rush. Depending where you were in the antebellum period, an ambitious young doctor would want to study in (in decreasing order of desiredness) Paris, Vienna, London, Philadelphia.
Even going into the late 19th century, Penn remained a powerhouse, along with the newly professionalized Harvard and Johns Hopkins in Baltimore, now explicitly modeled on the German medical university, rather than the English. The US continued to have a hodgepodge of medical training programs largely until the Flexner report was published in the early 20th century, which reorganized medical education on largely the Germanic model (though many universities, such as Penn and Harvard, as well as my alma mater, had already done this -- Germany had an outsize influence on American medicine during this period).
So, the TL;DR -- in the antebellum South, if you couldn't make it to continental Europe or to London, Philadelphia was your best option for a quality medical education.
BTW -- I'm currently reading Bliss' biography of William Osler; this is all in a slightly later period (1870s), but Osler, an ambitious young man from the Canadian backwoods, had a similar journey -- starting at McGill as it was professionalizing, then down to Penn, and then to Johns Hopkins where he would become legendary (before finally going to Oxford). It's a great book if anyone is interested in probably the most formative figure in American medicine!