In 1585, the Jesuit missionary to Japan, Louis Frois wrote:

10a. Because of our buttons and lacings, we cannot easily keep our hands close to our bodies; since Japanese men and women are not thus restricted, they always leave their sleeves hanging empty and pull their hands in close to their bodies, especially in winter.

• p. 41. The First European Description of Japan, 1585: A Critical English-Language Edition of Striking Contrasts in the Customs of Europe and Japan by Luis Frois, S.J., edited by Daniel T. Reff, Richard K. Danford, Robin D. Gill.

The editor's annotation for this passage:

With minimal heating (see Chapter 5 on houses) and no wool to speak of, the Japanese understandably conserved body heat by withdrawing into their kimonos. Frois* Jesuit contemporary, Rodrigues, noted that this withdrawal also was useful in hot weather, as"... people can insert their hands inside with the greatest of ease and wipe away body sweat with a handkerchief's However, to do so in the presence of nobles, one's master, or on formal occasions was a grave discourtesy and impertinence. Today on television "Eastems," loose-dangling sleeves are a mark of the gambler and other social misfits.

So, historically, people had their arms inside their kimono for comfort, and there is lots of evidence to show that many early modern Japanese people wore their clothes very loosely, and would tuck up or pull down/open their kimonos as convenient, with no worry of indecency.

But empty sleeves as a sign of attitude is a separate issue. I don't know when this association was made, but the figure of the gambler with one hand inside his kimono can be found in mid 19th century ukiyo-e. Here's Utagawa Kunisada's The Gambler: Kikunosuke .

There's one iconic photo from the end of the Edo Period of a famous samurai (and sometime ronin) Sakamoto Ryoma affecting this style. He appears to have his left arm inside his kimono, leaving the left sleeve empty, and his right arm (in its sleeve) reaching into his kimono breast. His hands are probably clasped together. Sakamoto's photo is a rarity, since most contemporary photographs feature people meticulously dressed and posed. Sakamoto's pose is certainly deliberate, part of his persona: an unruly ronin, rough around the edges but passionate for his cause. He had an image he was presenting, and the same image of the rough ronin with the empty sleeve has continued in more modern media.