Herodotus indicates, in his Histories (c.430 BCE) that the casing stones were present and had inscriptions of strange characters on them. He doesn't seem to mention the presence or absence of a cap/capstone, which suggests that something may have been there but that it may not have been the original--assuming that we take his account as eyewitness and not merely a traveler asking about objects in the distance. If the casing stones were pitted or worn, cracked, or what have you, that seems to have not been noted.

Diodorus Siculus 370-ish years later (60 BCE?) noted that some fragmentary stones laid strewn around the bases, but that most were in place, save the apexes. If they're Romans, they'd be closer to this era. It's noteworthy that the third pyramid's casing was supposedly half darker granite, not limestone, per William Flinders Petrie's (really old, and deeply Orientalist-tinged, but detailed) notes in The Pyramids and Temples of Gizeh from his fieldwork ca. 1882; this may be the "Black stone from Ethiopia" that Strabo in his Geography mentions in 20 BCE as covering the lower half, but Pliny's Natural History in 20 CE claimed that it was covered in marble then.

As to when the casings vanished, that's a matter less certain, and it may have [e: probably did, really] happened over an extended period. Flinders Petrie in his History of Egypt states that the 12th century saw the cannibalization of various stones from the pyramids for some construction in Cairo, but for the casings (see p.203). I see the earthquake of 1303 cited as a key moment for loosening many stones but also destroying or damaging many buildings in Cairo, and Sultan an-Nasir's construction of mosques up to the 1350s (which incorporate many of these stones) as a key culprit, with signature works of later rulers of Egypt as further collecting these stones.

WFP's Pyramids and Temples (217) of course channels the old canard of Oriental Despotism in characterizing the destruction of the casing ("The history of the destruction of the Pyramids really begins with the Arabs") which should urge a big grain of skeptical salt in taking him at face value. It's also worth noting that Britain had effectively just taken control of Egypt (in all but name) when Petrie did his work and published it, so he's operating as a cataloger of conditions that British civilization would investigate and arrest, in a mould not wholly unlike Napoleon's encyclopedists. Imperial reach as a mode of salvation from cultural degeneracy for "shared heritage" (Biblical or Classical) is a common theme of the 19th century, from this to the Elgin Marbles and everywhere in between.

More recent chronologies may differ on the timing and extent of teardown of the casings, but I'd suspect that the presence of the stones at the end of the Ptolemaic era is believable--if only because the importance of the Pharaonic position would have precluded dismantling more so than, say, the Roman or Caliphate (or later) rulers who had no investment in that line of legitimacy. The golden apex may be the stretch, and the condition of the stones was not likely to have been perfect--many could be chipped, cracked, and broken, and a few perhaps removed (as noted by visitors).

Please, Egyptologists [and Classicists of all stripes], if you've got something more recent than an imperial archaeologist to provide fine grain of the casing's loss, bring it here and correct me! I'd love a more recent omnibus to reference. After all, a lot of people seem to think the Turks alone ground it up for mortar or gunpowder after 1517, because that's what a bygone era would expect the "unspeakable Turk" to do--but the reality is clearly more complicated even by the Victorians' own reckoning.

[edit @ 7 mins - As a bit of background: the reason I go to Flinders Petrie is because his world is the one I know--he's an imperial scientist who corresponds a lot with those who work in Africa in the same era, which is also the era/region/subject I work on as a historian. But this question was one of many in the era of Biblical and Classical archaeology that many dwelt upon in the late 19th century. He was, like almost all such figures, deeply prejudiced and invested in Empire to one degree or another, which has various effects on their work that we must read around.]