EDIT: I realise I phrased the opening to this very strangely. Clarified and corrected.

I'm not personally familiar with the work of McCullough I must admit, though I'm vaguely aware of who he is, but he is quite right to claim that John Adams himself was not a slave owner. He was certainly not the only non-slave owner, though. His claim might depend on how we are defining the term 'founding father' - historians do not work with any single definition of the term, and it may be that McCullough is using a particularly narrow definition. I cannot think of a means of limiting the definition of 'Founding Father' that would include John Adams whilst excluding other non-slaveowning founding fathers, though - people like Samuel Adams, Thomas Paine or Roger Sherman. I would suggest that McCullough has most likely either used an unusual and uniquely selective criteria for determining who should be a Founding Father or, more likely, is misrepresenting the truth (or has not done his research properly).

So yes, I would say that it's fair to say he's off-base with his claim and I would be concerned if it's one that he has consistently made. I would also be critical of the context in which he is making the claim - it is important to remember that being anti-slavery in this period is not the same thing as being an abolitionist. It is possible for an individual to hold moral opposition to something without believing that its immediate end is necessary. John Adams was opposed to the institution of slavery on moral grounds but saw immediate and total emancipation as a disaster for all involved, and he supported at best a programme of gradual abolition of slavery, in which slavery would be phased out. In part, this stems from the fact that Adams had a mildly rose-coloured view of the institution, believing that many slaves lived happier lives than many poor white workers did, and feeling that the institution was in sharp decline. It was not a uniquely pressing issue in his mind. You can read his own words on the topic from an 1801 letter, which I have reproduced substantially below:

Although I have never Sought popularity by any animated Speeches or inflammatory publications against the Slavery of the Blacks, my opinion against it has always been known and my practice has been so conformable to my sentiment that I have always employed freemen both as Domisticks and Labourers, and never in my Life did I own a Slave. The Abolition of Slavery must be gradual and accomplished with much caution and Circumspection. Violent means and measures would produce greater violations of Justice and Humanity, than the continuance of the practice. Neither Mr. Mifflin nor yourselves, I presume would be willing to venture on Exertions which would probably excite Insurrections among the Blacks to rise against their Masters and imbrue their hands in innocent blood. There are many other Evils in our Country which are growing, (whereas the practice of slavery is fast diminishing,) and threaten to bring Punishment on our Land, more immediately than the oppression of the blacks. That Sacred regard to Truth in which you and I were educated, and which is certainly taught and enjoined from on high, Seems to be vanishing from among Us. A general Relaxation of Education and Government. A general Debauchery as well as dissipation, produced by pestilential philosophical Principles of Epicurus infinitely more than by Shews and theatrical Entertainment. These are in my opinion more serious and threatening Evils, than even the slavery of the Blacks, hateful as that is.

I might even add that I have been informed, that the condition, of the common Sort of White People in some of the Southern states particularly Virginia, is more oppressed, degraded and miserable than that of the Negroes. [source]

So you should certainly be cautious if the comment about Adams being the only non-slaveowning Founding Father was made as part of a bid to try and present him as a uniquely profound and moral figure in the historical record. Like many anti-slavery figures of his time, his views were complex and coloured by contemporary racial politics and cultural attitudes. It may be that McCullough is being a little selective with how he presents his material in order to 'big up' a president he has particularly affinity for, which is sadly not unusual in populist biographical history of prominent political leaders.