In the Quran, Mohammed's family are referred to as "Hanifs" - members of a sect of monotheists, who were neither Jewish nor Christian. What were the beliefs of these Hanifs, and how were they different from Jews and Christians?

Great question, one for which we do not have a certain answer! So instead I'll tell you some interesting information which may point to an answer. The word hanif in Arabic is related to a word hanpa/hanfa in Syriac, which was a (predominantly Christian) dialect of Aramaic (i.e., the language Jesus spoke) and the lingua franca of the Near and Middle East until probably 1000AD or so. Syriac-speaking peoples have long had contact with Arabs, including before the Islamic period, and their texts give us some of the best insights into early Islam. (See, for instance, Michael Penn, When Christians First Met Muslims, but there are a ton of books on this.)

So this should solve the problem, right? We just need to see what this means in Syriac. Well, it turns out to be more complicated, because in fact the word hanpa is curiously the Syriac word for pagan! How did a term of opprobrium in Syriac become a positive word for the monotheistic qualities of Christians and Jews? Hanpa and consequently hanif must have had a broader semantic range (range of meaning) than simply "pagan." Rather, in Syriac it probably encompassed the sense of Gentile -- that is, a non-Jew, but of course the term Gentile acquires a more positive meaning in Christianity, since Jesus commands his followers to make disciples not only of the Jews but of the Gentiles.

So what does this word mean in the Qur'an? Pagan? Gentile? In fact, it seems to take on a new meaning, meaning something like a "righteous, Abrahamic monotheist." Abraham is the paradigmatic hanif. Q2:135 is very clear about this: "The Jews and Christians each say, 'Follow our faith to be rightly guided.' Say, O Prophet, 'No! We follow the faith of Abraham, the hanif...'" So hanif does not only mean a kind of pre-Abrahamic devotion to God (which elsewhere in the Qur'an is referred to as the paradigmatic islam, that is, submission) -- hanif is also an integral part of the Qur'an's polemics against Christianity and Judaism, to the effect that those groups have fought each other and corrupted their sacred texts so that what they believe is no longer the pure monotheism and submission of Abraham, but something much different and decidedly less good.

Further Reading: Cole, Juan - "Paradosis and Monotheism: A Late Antique Approach to the Meaning of Islam in the Qur'an" de Blois, Francois - "Nasrani and Hanif: Studies on the Religious Vocabulary of Christianity and Islam" Dye, Guillaume - "Traces of Bilingualism/Multilingualism in Qur'anic Arabic"