The soldiers of the U.S. Colored Troops were frequently sent to hostile environments rife with disease, largely due to the belief that blacks suffered less from yellow fever, malaria and other tropical diseases than whites did. Exacerbating this problem was the fact that many black enlistees were in poor physical health before they entered the U.S. Army. Any health problems they may have had were aggravated by the fact that black soldiers received poorer supplies, including food, shelter and medical care than white soldiers. Black soldiers were worked hard in entrenchments and on construction projects, many sickened by such labor in unhealthy conditions. Furthermore, even minor wounds can become infected and fatal without attention.

The Sanitary Memoirs of the War of the Rebellion seem to be the source behind the link you mention, but there is other data available. Fortunately, Volume 1 of those memoirs is posted online, and if you turn to page 184, you'll also see the 1867 theory as to why the proportion of black soldiers dying from disease was so much higher:

"These figures appear to indicate pretty conclusively that the negro, as he was found in our armies, was less capable than the white man of enduring the fatigues and hardships, and of withstanding the influences of disease incident to army life. ... What the causes of this greater susceptibility of the negro to disease are, hardly come within the scope of this paper; but it is suggested, to use the words of General Fry, 'that they were rather psychological than physical, and arose from lack of heart, hope, and mental activity, and that a higher moral and intellectual culture would diminish the defect.'"

Of course, Gen. Fry's words are so much bunkum.

James K. Bryant's The 36th Infantry United States Colored Troops in the Civil War concludes, "The high death rates of disease among black soldiers, particularly of the 36th USCT, resulted from poor pre-war physical condition, excessive fatigue duty in constructing trenches and other military fortifications, and poor medical care."

Remember, the men enlisting in the USCT were often escaped slaves and not the pinnacles of health that we associate with outdoor labor today. They were malnourished, and physical examinations "revealed shocking signs of physical abuse received in slavery including scars from whips, burns, and mutilation."

There were few black doctors to serve in black units, and white doctors frequently refused to serve in black units or treat black soldiers. This meant that the USCT was chronically understrength in surgeons and orderlies, people who typically enforced sanitation orders in camp. Units with less care to latrines and clean drinking water could suffer high rates of dysentery and other waterborne diseases.

These poor conditions were not just present at the front or in battle camp. An October 1864 report on a camp in New York states: "For a considerable time the quarters provided for the colored men were insufficient and improper. Tents were furnished by the Government, but ... the men were greatly crowded; they were also without floors or means of warming, causing great suffering from cold. Disease began to appear to an alarming extent, while there was no proper hospital in which to treat it."

Late in the war, adjutant general Lorenzo Thomas visited a USCT hospital in Nashville and found lice-infested wards with men still wearing bloody uniforms from battle. "Words cannot describe the utter filthiness of what I saw," he said. "Had these men been white soldiers, think you this would have been their condition? No!"

Racism was present from coast to coast. In the anthology Black Soldiers in Blue, John David Smith touches on the case of the 56th USCT in Arkansas. Limited to garrison duty, it lost 25 officers and men in combat — and 653 men to disease. Smith refers to the historian Ira Berlin, well-known for African-American and Southern history, who describes how Southern black soldiers sometimes would wait around white cookhouses to get food left over from white soldiers better-fed than they were.

As for disease, you'll want to refer to Andrew Mcllwaine Bell's Mosquito Soldiers, a pretty good breakdown of the impact that malaria and yellow fever had on the course of the war.

Apart from the excellent answer below, it is also important to consider that the majority of deaths in all units, of all races, North and South, were from disease.