A number of Scottish Masons were hired for the purpose. They had developed a number of specialized techniques that were designed to protect the kind of stone selected for use from water. James Hoban, an Irish Architect, styled it after a Ducal Palace in Ireland, which now happens to be where the Irish Parliament sits. So, using masons familiar with the processes used there seemed to make a great deal of sense.
That said, there were five slaves employed as carpenters in a list of workers on the White House project in 1795, three years into the eight year process. It's also probable that much of the quarrying and logging also included some portion of slave labor. Some records seem to indicate that the government itself owned some slaves at a quarry, the stone of which was used at both the White House and the Capitol Building.
So, the overwhelming majority of labor used on these sites were standard issue construction workers, but there was a smattering of slaves mixed in. To say that it was built by slaves might be misleading depending upon the context.
Edit:
According to the documentation provided by the White House Historical Association it was a case of the government hiring slaves owned by private citizens. There were a handful of slaves owned by the Federal Government kicking about in this time period, but none of them were used in the construction of the White House.
The Invisibles: Slavery in the White House and How it Shaped America is an interesting read.
Another, but only tangentially related one that is fascinating is A Slave in the White House: Paul Jennings and the Madisons. Or maybe just read Jenning's own book: A Colored Man's Reminiscences of James Madison published in 1865.
Sorry to be so late to the party, but I actually answered this earlier on my blog, Rethinking the Revolution. I'll copy and paste my answer here from there, but I also included a few pictures with it as well for those of you who may be interested.
Short Answer: Both the original White House and the one which was rebuilt after the 1814 attack were largely built by enslaved Americans.
Long Answer: While I generally don't like or want to talk about politics on my blog, when politicians invoke the founding during their speeches, I believe people have a right to question their usage. On Monday, during the first day of the DNC, America's First Lady, Michelle Obama, said, "I wake up every morning in a house that was built by slaves," and one viewer asked if this was true. So here we go.
Black Americans, many of whom were enslaved, are deeply tied to the creation of the White House. As one historical journal put it, "The story of blacks and the White House begins with slavery. Black slaves were instrumental in the building of the White House."[1] That same journal even points out that the second child born in the White House was actually to an enslaved woman who was owned by Thomas Jefferson. Slave ownership by the American gentry was common during this era -- after all, 12 American presidents owned slaves, so it should come as little shock that they were used in the creation of the nation’s capital.
Pierre Charles L'Enfant Pierre Charles L'Enfant began designing Washington, DC in 1791 and, from the very start, American slaves were leased from locals in Maryland and Virginia to begin clearing the sites of the city, including the area where the President's Mansion would be erected. However, this wasn't the plan from the outset. Originally, architects and designers planned to employ workers from Europe to do the digging and building, but when they were unable to secure these workers, they had to turn to other means of labor. Between 1795 - 1800, 122 names appear on wage rolls listing them as "Negro hire" which is 18th century jargon for "slave." Additionally, it is important to note that some free black Americans and white workers also were involved in the White House’s construction.
The White House Historical Association's own website actually explains it well when they said:
Stonemason Collen Williamson trained enslaved people on the spot at the government's quarry at Aquia, Virginia. Enslaved people quarried and cut the rough stone that was later dressed and laid by Scottish masons to erect the walls of the President's House. The slaves joined a work force that included local white laborers and artisans from Maryland and Virginia, as well as immigrants from Ireland, Scotland, and other European nations.
Other specialty roles were also filled by enslaved Americans. For example, a carpenter's role dating to 1795 showed four enslaved people, named Tom, Peter, Ben, and Harry, who had worked on the building during that time. All four of them were enslaved.
This also doesn't count the dozens (perhaps even hundreds) of enslaved men who were forced to clear the forests and drudge out the swampland that surrounded the immediate areas around the White House. There's no way to deny it, First Lady Michelle Obama was not using hyperbole -- the President's Mansion was quite literally built by Americans who were enslaved.
Source:
Would this be the correct thread to ask a follow-up question? Bill O'Reilly of Fox News stated; ""Slaves that worked there were well fed and had decent lodgings provided by the government, which stopped hiring slave labor in 1802."
Where did he get this information and is this accurate?