In the 1910s and 20s, most record buyers would have bought records from the same places they would have been buying sheet music for the past several decades. That is, they would have bought records from five-and-dime stores such as Woolworth's, or else from the mid-market and upscale department stores such as Macy's, or else from dedicated "music houses" that sold not only records and sheet music, but pianos and trombones. As some examples:

• A 1905 ad for the Collins Piano Co. in Omaha, Nebraska, offered "Talking Machines" (a.k.a. record players like Gramophones and Phonographs) and "Records" for sale, among its products.

• A 1909 ad for the Junius Hart Piano House in New Orleans listed "Pianos, Organs, Piano-Players, Talking Machines, Records, Sheet Music and Musical Instruments" as its offerings.

• A 1910 ad for Hegglund's Music Store in Pierre, South Dakota, promoted itself as "the cheapest place to buy Pianos, Edison and Victor Phonographs, Records, Sheet Music, Type Writers, etc.".

• A 1915 ad for Knight's Music House in Perth Amboy, New Jersey, claimed to sell "Everything Musical" including "Phonographs, Records...Sheet Music...Record Cabinets" and more.

• A 1918 ad for Warren's Music House in Pendleton, Oregon, asked, "Is there MUSIC in your HOME?" while promoting their "complete stock of everything musical" including the "Latest Records".

One interesting outlet for record sales that seemed to be relatively common at the time was jewelry stores. Evidently, some jewelry stores had expanded their businesses to include the sale of proto-"electronics", like sewing machines and typewriters. Record players must have seemed a natural fit, and with them came a stock of records. Jewelry stores in Birmingham, Alabama, in Wildwood, New Jersey, and in Oakley, Idaho, as some examples, listed records as among their merchandise. The Birmingham store was so successful, in fact, that they moved their jewelry business to a separate building, while the original building became dedicated to their music business.

One other option to purchasing records in the 1910s and 20s, as you suspected, was mail order. None of the record labels operating in that period ever sold records directly to the public (though someone feel free to correct me if I am wrong). For instance, a 1914 advertisement for Columbia Records says that any local record "dealer should be able to supply" Columbia Records to the interested reader. No mention of a catalog, mail order, or otherwise buying records directly from the company, though there is a coupon for a record if you send in the name of your "nearest Columbia dealer".

But many local and regional department stores did have mail order businesses, so mail order certainly was an option for record-buyers. For example, The Corley Company in Richmond, Virginia, was offering mail order record sales at least as early as 1912. The company had been spun off from the local Cable Piano Company, which sold pianos and other musical instruments.

But probably the most common outlet would have been Woolworth's, or whatever the local five-and-dime chain happened to be in the area. While focused on sheet music sales, David A. Jansen writes in Tin Pan Alley: An Encyclopedia of the Golden Age of American Song:

The biggest sales of sheet music were in five-and-dime chain stores (Woolworths, McCrory’s), where a sheet sold for ten cents. At a music store (which also sold musical instruments, phonographs, records, and piano rolls), the retail price was twenty-five cents during the 1920s, rising to forty cents after World War II.

While Woolworth's and other five-and-dime chains didn't do a whole lot in preserved printed advertising, they apparently didn't have to. For example, testimony from a 1914 court case in New York included a passage where the witness had hoped to make a half million dollars by getting a record onto the Woolworth shelves, selling them for ten cents a pop. When Billboard began making its earliest sales charts (first for sheet music, later for recorded music), the numbers were based off of reported sales at several five-and-dime chains--including Woolworth's--as well as major department stores and "music houses".

The larger, "flagship" (so to speak) department stores around the country were also a prime mover in record sales. For example, Hudson's in Detroit was selling enough records by 1914 that they leased an entirely separate building just for their music business, which remained open until 1931. The department store built a new, larger (massive) building that opened its doors in 1927, which was the second-largest in the country, by square footage, after Macy's. A few years after that, then, the music business was moved into this new building, and occupied the entirety of the 13th floor. That one floor was quite possibly the largest "music store" in the country for the next couple decades, until the company started shuffling things around as the department store era began its decline in the 1950s.

Actual dedicated record stores had begun to appear no later than the early 1930s. George's Song Shop in Johnstown, Pennsylvania, claims to be the oldest still in continuous existence, first opening in 1932. But I am unsure what exactly distinguishes these stores from the earlier "music houses" which sold instruments and sheet music, too. It appears that many of the "oldest record stores" that still exist were initially "music houses" that came to focus on recorded music sales during the 1930s and after, when radio had helped recorded music supplant sheet music as the king of music sales. Some of the dedicated "record stores" were, no doubt, initially in the business of selling sheet music and musical instruments.

To go a bit further forward in time, the "five-and-dime"/discount retailer importance of record sales continued into the late 1950s and early 60s, when "stereo" record players were first introduced. By then, these stores were competing with specialty "record stores", but it didn't stop them from being a major player in spreading the new technology. According to The American Popular Music Business in the 20th Century by Russell Sanjek:

More than 2,500 outlets joined the [stereo record player] boom, when Newberry, Woolworth, Kress, and J.C. Penney added low-price stereo players, bearing their own names, to their stock.

You can still find some of these vintage stereos for sale on eBay occasionally. For example, here is one branded by J.C. Penney.

In closing, to reiterate, if you wanted to buy a record in the 1910s or 20s, you probably would have headed to the nearest Woolworth's, McCrory's, Kresge's, or whatever your local five-and-dime store was. If not there, then you were likely going to Macy's, Gimble's, Hudson's, Wanamaker's, Filene's, Marshall Field's, or whatever the major department store in your city happened to be. Beyond that, then you were probably going to the "music house" that also sold sheet music and musical instruments.

I believe phonograph recordings, early Victrola, and the like were often sold at furniture stores because the devices that played them were then considered furniture. That's how Paramount started, as a furniture company based in Wisconsin, but quickly expanded the music wing of their operation to meet consumer demand, especially where the ever expanding race-record market was concerned. That's why Blind Blake is buried in Wisconsin rather than Florida. He was doing recordings for Paramount and died during the period of his last sessions. That's also how a bunch of vintage acetates ended up at the bottom of a river in Wisconsin; when the original Paramount furniture company went belly up they chucked a bunch of them in the river as they were clearing out the inventory. That was a hell of a find years later for American historians and historical music enthusiasts, alike.