The act of painting fingernails was a common practice in Egypt by 3000 B.C. but it is believed that fingernail paint originated even earlier in China where the color of your nails showed what your social rank was.

By the third millennium B.C. the Chinese used ingredients such as gum arabic, egg white, gelatin, and beeswax to make varnishes, enamels, and lacquers. The color nails that determined you were royalty, according to a fifteenth-century Ming manuscript were black and red. Although in the Chou Dynasty of 600 B.C. gold and silver were seen as the royal colors.

This was also true with Egyptians where the color of your nails signaled what your social standing was. In Egypt red signaled great importance, Queen Nefertiti painted her fingernails and toenails a ruby red, Cleopatra favored a deep rust red. Women who were of lower social class were only allowed pale hues, no woman would risk having the colors that a queen had.

Painting nails was also common for warriors of great importance. Egyptian, Babylonian and early Roman commanders would spend great amounts of time having their nails painted the same shape as their lips.

The great amount of attention ancients showed to fingernails and toenails suggests to most cosmetics historians that manicuring was already an established art. This is supported by excavations of royal tombs at Ur where manicure sets were found. "Well-manicured nails became a symbol of culture and civilization, a means of distinguishing the laboring commoner from the idle aristocrat".

Panati, Charles . "Atop the Vanity." In Panati's Extraordinary Origins of Everyday Things. New York: Harper & Row, 1987.

If you're interested in nail polish, you're on the right track with the automotive paint thing. Clear nail polish was invented in the 20s, and colored nail polish developed in 1930. From "COSMETICS AND SKIN CARE PRODUCTS : A Historical Perspective":

Nail enamel or polish is useful for nail adornment, covering nail discolorations, and providing strength to weak nails. Nail polish was introduced in the 1920s when lacquer technology was developed. During World War I, excellent sources of nitrocellulose were developed as a military explosive. The nitrocellulose was created by reacting cellulose fiber, from cotton linters or wood pulp, with nitric acid. It was discovered that boiled nitrocellulose could be dissolved in organic solvents. After evaporation of the solvents, a hard, glossy film of nitrocellulose was produced, known as a lacquer. Extensive research on nitrocellulose lacquer was undertaken by the automobile industry, which found the product preferable to slow-drying, oil-based paints previously used to paint cars. This technology was adapted directly to the cosmetics industry.

Before 1920, nails were manicured, then polished with abrasive powder to achieve a shine. Color was then added through the use of stains. The first lacquer marketed was clear and labeled a nail polish because it imparted a high shine to the nail plate. In 1930, Revson developed the idea of adding pigments to the clear lacquer to form an opaque, colored nail polish. Based on the success of this poor-quality nail polish, Revson formed Revlon in 1932. He hired a formulation expert to develop a better product, which became known as nail enamel.

However, as they note, women were coloring their nails long before the invention of nail polish. If you're interested in when any humans anywhere began to paint their nails, the answer is going to be in prehistory. But if you're interested in its rise in Western European countries or something along those lines, that's findable, I'm sure.

The Chinese were painting their nails as far back as 3000 BC, although specific class-based color identification (gold colored nails reserved for royalty, for example) was first recorded during the Zhou Dynasty (1046-256 BC). By the time of the Qin dynasty (221-206 BC) red (or black) seems to have replaced gold as the color intended for use by royalty/ nobility (Sherrow, 2001). It's interesting that social class restriction to nail colors were not just limited to ancient China, but also to places like Egypt, which suggests some universality of the concept of social ranking by nail color.

Of course, the idea of class identification by fashion styles isn't just limited to nail color, almost every item associated with fashion at one time or another was also associated with social status (for example ermine cloaks for royalty) or caste delineation (for example the "sacred thread" of Hindu Brahmins). Some folks have gone as far as to suggest that the true symbolic purpose of fashion is to register and de-register individuals with respect to distinctions of class and social status (Davis, 1992).

Sherrow, Victoria (2001). For appearance' sake: The historical encyclopedia of good looks, beauty, and grooming. Phoenix: Oryx Press.

Davis, Fred (1992). Fashion, Culture, and Identity. Chicago: The University of Chicago Press.

Edit: I wonder if folks were painting their nails for functional reasons rather than cosmetic -- for example, musicians to strengthen their nails for strumming string instruments. A quick search didn't reveal any meaningful hits, maybe someone on this sub knows if nail-painting is purely a cosmetic activity or if there other historical + functional uses for painting nails.

This is a good ebook about the history of cosmetics, only goes back to the Victorian era though http://www.pen-and-sword.co.uk/The-History-of-Compacts-and-Cosmetics/p/6885/