We refer to the decades of the twentieth and twenty first centuries as being the "twenties" etc, but we don't have any such terms for times before 1920.
Is this because we're so far removed from that time we look at the period in relation to what happened, rather than each decade (i.e. it's the period between this war and that, or the industrial period, or the renaissance), or because there was no cultural habit of counting the decades like we do now before the 20s?
Would an elderly chap in 1898 reminisce over how good things were back in the 60's?
I hope I've made this clear enough.
As always in the case of "when was [word] first used?" questions, you should always start by checking the Oxford English Dictionary (the OED, for short). In case you don't know, the OED gives a number of attestations for all the senses of all the words they list. The earliest listed example is supposed to be the first known example of a word being used in a particular way, at least as far as the OED people can find.
I checked a bunch of these words in the OED (in the sense "the years between X and Y in a particular century") to get an idea of how old these terms were. Interestingly, they're almost all from the 1880s.
Examples:
Twenties:
1886 J. R. Seeley Short Hist. Napoleon I 262 Had Louis XV died in childhood..there would certainly have been in the twenties a war of the French Succession.
Thirties (same guy, interestingly):
1883 J. R. Seeley Expansion of Eng. 288 Dating only from about the thirties of the present century.
Forties:
1885 Athenæum 18 July 83/1 His magnum opus was published in Edinburgh some time in the forties.
Fifties and sixties (earliest for both of them):
1889 R. B. Anderson tr. V. Rydberg Teutonic Mythol. 9 A series of works published in the fifties and sixties.
Seventies:
1887 W. E. S. Fales Brooklyn's Guardians iv. 57 Before the ‘Seventies’, Brooklyn was essentially a city of homes.
Eighties (this guy, again!):
1883 J. R. Seeley Expansion of Eng. 260 Adam Smith, writing in the eighties.
Nineties:
1853 Littell's Living Age 17 Dec. 766/2 In a few more years, the men of the nineties will be in course; and so it goes on.
A few notes:
The OED is not all-knowing, it's entirely possible that they've missed earlier examples.
Almost all of these words had earlier examples where the word was used for a particular period in someone's life (i.e. "When I was in my twenties...").
I couldn't find any senses in the OED for words describing the first two decades of a century (i.e. the "aughts" and the "tens"), though perhaps I didn't look hard enough.
The first time someone wrote something down is obviously not the first time a word was used like that, it could have been used in colloquial speech much earlier than that.
Fun fact: the first time someone used "teen" to indicate someone's age was in 1664. Way older than I would've guessed.
Given that the earliest example is from 1853, I think it's reasonable to assume that this usage originates in colloquial speech some time in the first half of the 19th century.
The Gay Nineties is actually a term for the 1890s which was quite common in the early twentieth century (although I imagine it started later than the actual 1890s, as it's obviously a term of nostalgia). I still see it sometimes on collections of sheet music. I'm not sure if that is where the custom of referring to decades like that comes from, or not. I'd be interested in hearing about it from modernists, if anybody here knows!