I'm unsure of the specific mechanics of sending would-be immigrants back, but I know for a fact that they did indeed get deported for medical reasons at the very least. Here is 1920 testimony from the Surgeon General to Congress about incoming immigrants indicating that of "536 cases of loathsome contagious or dangerous contagious diseases; there were 239 deported; and 167 landed" (here, I'm assuming "landed" means allowed to stay). Eventually the deportation of freshly arrived immigrants was enough of an issue that the US implemented examination of immigrants before they arrived. Per Cumming's biography here:

In 1925, Cumming inaugurated a plan for the medical inspection of immigrants abroad in the principal countries of origin. This plan reduced the number of immigrants who were turned back for medical reasons after making the trip to this country. In the 1920s, the PHS also completed the development of a national maritime quarantine system by acquiring the last two quarantine stations operated by States.

Regarding the mechanics of the whole thing, I would only be able to speculate. However, it seems it would be easiest for immigration officials if they simply did not let the immigrants in question leave their ship (if something like fake paperwork was caught before disembarking) or stick them back on the same ship if something was found at the immigration station (such as the health issues above). While it might seem unfair to the captain of the ship to burden them financially with passengers unable to pay, it would have prevented the captains of other ships or the government from footing the bill. Finally, even if someone is penniless, the captain could force the unwanted passenger to work for his or her passage shoveling coal, working in the galley (although hopefully not someone deported for having a contagious disease!) or performing other menial labor.