It should go without saying that all the links here are VERY NSFW
Did people use cameras in private for nudes of a sexual nature before the internet? In short, yes they did. The advent of photography in the nineteenth century saw a proliferation of nude photography which was often quite sexually explicit. While these nudes were done in various studios and not "homemade," the photographers often tried to sell the appearance of an intimate domestic setting or other forms of voyeurism such as a tryst in a confessional booth or various spanking photographs. This is not too far removed from some contemporary pornographic studios like Abby Winters that make a point to stress the naturalness of their settings. Not surprisingly, there was a rise in the personal documentation of sex and nudity as the costs of camera ownership fell in the first half of the twentieth century. The lack of any proper provenance makes difficult to tell if the surviving films and pictures from this period were the product of personal consumption or done in a studio, but just as in the earlier period, pornographers in some cases did make an attempt to make it look like a domestic or private environment (or shot the scenes in their own houses).
One hurdle for DIY pornography in the mid-twentieth century was the processing of film and in the US, various laws against the distribution of obscene materials through the mail system. However, many amateur enthusiasts found ways around this via camera clubs and claims that the nudes were not pornography, but art. This was the start of Bettie Page's modelling career, with her becoming one of the more popular glamour models in the NYC camera club scene. The "artistic" angle was one also employed in the burgeoning American gay subculture with male nudes marketed as celebrations of the male form or ethnography. There were also more explicit sexual images floating around the grey and black markets of this subculture. The midcentury diarist Donald Vining recalled that not only was the artistic merit of this media lacking, some were quite old:
After I migrated to New York in the early [1940s], I discovered that pornography of a very crude sort was circulating in the gay sub-culture. It seemed in fact, to have been circulating for a long time, perhaps decades. From the hairstyling and backgrounds of the pictures, it appeared that the models were Ruritanian bootblacks, lured from their cloud coo-coo land by the promise of two tickets to the Paris exposition of 1893. I imagined someone plying them with tokay to reduce their inhibitions about shedding their clothes. A bit too much tokay, indeed seems to have been consumed, or else the mushroom cellars where the photography seemed to have been done were very cold, for the cocks on view were so small as to resemble Vienna cocktail sausages. The quality of the photos suggested that the photographer also had perhaps hit the bottle too freely.
There was also a whole panoply of advertisements for these materials (straight and gay) in various low-quality publications. These advertisements often employed code words for more explicit materials, but despite these precautions, distribution of erotic material via the US mail was subject to prosecution, such as the case of Irving Klaw (one of the more prominent distributors of Page and other glamour pin-ups).
The development of Polaroid's instant camera in the late 1940s, which allowed for a quick DIY development of photographs without the need of a darkroom and its associated chemicals, meant there was a proliferation of nude images in domestic settings. Some of the more explicit Page photos came from the first-generation Polaroid cameras. The Kinsey Institute also had a number of Polaroids to document the expression of human sexuality and Betty Dodson based her charcoals documenting her sex life from Polaroid stills she and her lover took. By the 1960s, there were advertisements soliciting for nude Polaroids. As mainstream pornography became more explicit in the 1970s, one of the popular features of Hustler magazine was "Beaver Hunt" which advertised various "reader-submitted" photos (again, while the veracity of "Beaver Hunt"'s claim to be homemade is questionable, Hustler marketed "Beaver Hunt" as purely authentic). The Polaroid company was aware its cameras were being used in this fashion, but tended to approach the topic very gingerly. Polaroid's director of corporate communications Donald Dery summed the company's position in an interview, "we didn't acknowledge it, but we always talked about 'intimacy'."
So how was all of this different than the nude selfies and videos of today? While the internet and broadband has democratized and expanded the audience and production of erotic material, the content of such media is not terribly different from its forebears. For example, aside obvious markers such as hairstyles or period furniture, the poses and scenarios of vintage Polaroids is not much different than pics submitted to r/realgirls or r/amateur today.
Sources
Bonanos, Christopher. Instant: The Story of Polaroid. New York: Princeton Architectural Press, 2012.
Bright, Deborah. The Passionate Camera: Photography and Bodies of Desire. London: Routledge, 1998.
Waugh, Thomas. Hard to Imagine: Gay Male Eroticism in Photography and Film from Their Beginnings to Stonewall. New York: Columbia University Press, 1996.
Are you asking if nudes were popular, or pornography?