Yes, I think he is correct. But it is important to remember that they did not exclusively use Greek in private. Fronto was the tutor of Marcus Aurelius and a famous Latin rhetorician, but his letters to Marcus' mother were in Greek. Here he was acknowledging her as educated, and perhaps she especially enjoyed corresponding in Greek. Yet Fronto wrote to Marcus in Latin and even playfully scolded him for using Greek. This is an example of how they used Greek when it made sense, and didn't when it served no purpose.

Some may have been unable to speak Greek. Seutonius says that Augustus himself wasn’t confident enough to speak it extempore or to write his own speeches in it.

Also, some Romans looked down upon Greek. Juvenal, in his Satires, complains:

[Women] talk nothing but Greek... Their fears and their wrath, their joys and their troubles - all the secrets of their souls - are poured forth in Greek... All this might be pardoned in a girl; but will you, who are hard on your eighty-sixth year, still talk in Greek? That tongue is not decent in an old woman's mouth. When you come out with the wanton words ζωὴ καὶ ψυχή, you are using in public the language of the bed-chamber.

We need to be careful with Juvenal – it is a satire after all – but he shows that Greek was popular among the educated youth and was perhaps even the intimate language of love. Yet he also implies the use of Greek was almost like a ‘phase’ that one would grow out of.

So, yes, many of the Roman elite used Greek privately amongst themselves. But this depended upon the context of the communication and the social situation. The use of Greek was not ubiquitous among them.