It's an 80s thing. The Oxford English Dictionary suggests that the first use of the term "phone sex" was in 1982, in an article on the associated press newswire. Both the term and the concept appear to have been an invention of pornographic magazine publishers. Here's a snippet of the AP story to back that up:

NEW YORK (AP) - 21 Jul 1982

Dial-a-Prayer, Dial-a-Joke and other prerecorded telephone message lines have been joined by an X-rated newcomer named ''Free Phone Sex'' whose callers range from curious youngsters to bored night-shift workers.

''We're averaging 2,000 calls an hour,'' many of them long distance, said Ira Kirschenbaum, vice president of High Society magazine.

The call-in line is designed to bolster sales of the magazine, which features pictures of naked women in various sexual poses and is described by Kirschenbaum as ''strictly a girlie book.'' The prerecorded, three-minute ''message'' is an audio accompaniment to a series of photos in the monthly magazine that illustrate a prurient story line.

From there, it's pretty easy to see how someone would come up with the idea of making money off it - initially by charging a credit card for a "live chat." The earliest reference to phone sex that I could find in a "legit" newspaper is from the Chicago Tribune in 1983, which suggests that:

The unholy alliance of capitalism, technology and lust has made it possible - for the price of dinner and a move - for anyone with a credit card to patronize one of America's growing number of telephone bordellos. The mechanics are simple: For about $30, the customer hires an anonymous partner to "talk dirty" over the phone for up to an hour.

The Tribune points to the same line, run by High Society, as a pioneer, and notes that the cost of the calls (not the service) made the telephone company that carried them $150 000 in a week. It also quotes the publisher of Screw magazine ("the publication of record on such matters") as suggesting that "telephone sex has spread to big cities all over the country in the last several years and become 'tremendously lucrative.'"

The same article also gives a pretty hilarious rundown of how it worked:

Most ear sex works the same way: The customer calls the number listed in an advertisement and speaks to a receptionist, who takes the customers name, credit card number, telephone number and, in some cases, address and drivers license number. The customer hangs up while the receptionist runs a quick, standard credit check. The receptionist then contacts the person who will place the call and do the dirty talking... and gives them the customer's phone number. Long distance ear sex calls are placed collect. Most services promise 30 minutes or more of bawdy banter, but, they say, most men require less than 15 minutes.

Edit: Further research suggests that the High Society line was what they called a "Dial-It" number, a special line "only recently made available to private entrepreneurs" (by AT&T) which allowed the company running the line and the phone company to split the revenue on each call, which worked out to 15 cents for each local dial-in. A second line that they set up in LA cost 50 cents per local dial-in, and earned the phone company 19 cents and the magazine 31 cents per call - which was super profitable because High Society was using a recording rather than a "live chat" model, and they ended up getting a lot of calls from teenage boys (much to parents' chagrin).

That's easy. Everyone in the business knows phone sex was pioneered by Gloria Leonard, who was the first female publisher of a 'skin' magazine. She came up with the idea of guys paying to listen to recorded messages of the magazine's featured models. Made a fortune doing that. Then they kicked it up a notch having the guys talk live to the models at a slightly higher rate. That didn't work out so well. Not everyone can handle being an operator which is more than a nice voice and an open mind. That's when they came up with the current model.

It's worth noting that it wasn't until the internet that 'phone sex' became a stand-alone business (otherwise the only advertising was on the back of skin magazines) which is why you still see cross-over of model images etc.

Ms. Leonard passed away earlier in the year and AVN provided an excellent overview of her contribution to society including her role in starting what has become a multi-billion dollar industry. It's worth a read.