For most of American history, the media was openly partisan -- many liberal, many conservative. That began to change dramatically around the beginning of the 20th century, as publishers like William Randolph Hearst and Joseph Pulitzer found they could skyrocket their circulation (and profits) by printing news that appealed to people of all political persuasions. Other publishers followed their example. But as late as 1940, open partisanship was still common: newspapers reported their affiliation to "Editor and Publisher Year Book" that year as follows: Republican 472, Democrat 489, independent 887. In other words, one-quarter were openly liberal, and another quarter were openly conservative.

Increasingly, though, impartiality became part of the professional ethic of journalists. This partly made business sense -- you can have a much bigger audience if everyone trusts you're telling it straight -- but also became, as I said, a matter of ethics for journalists who thought (and think) that unbiased, accurate reporting is essential for democracy. This was also mandated by law starting in 1927 for the radio and TV broadcasters from which more and more Americans got their news: Since the radio waves were public, broadcasters were required to give "equal time" to all candidates, and from 1949 to 1987 were required to present all matters of public importance fairly, as judged by the Federal Communications Commission. (Note that it was a Republican president, Ronald Reagan, who pressured the FCC to kill the rule.)

When people today speak of "liberal media," it's often implicitly contrasted with a golden age of fairness. That age isn't necessarily defined, but one common example is Walter Cronkite, "the most trusted man in America." Cronkite was a CBS anchor who, according to a survey taken around the time of Watergate, was listed as trustworthy by 73% of respondents, more than any other public figure (second was the generic "average senator" with 67%. Clearly, Americans were much more trusting of public figures in general in 1972 than they are today.) CBS marketed that heavily, and it sticks in the minds of many people today who can't actually say why Cronkite was trustworthy but modern newscasters aren't. But Cronkite also famously did an editorial segment concluding that, in his considered opinion, the U.S. could not win the Vietnam War. In his autobiography, Cronkite recalls, "I am proud to the degree to which we kept our evening news free of bias, although on a subject as controversial as the war, we did not get credit from either side for doing so. The conservatives and government supporters thought we had joined the wild-eyed, 'unpatriotic' liberals. The students and other opponents hounded us as mouthpieces for the establishment." In my estimation, in the modern era, these two somewhat-contradictory criticisms have almost always been lobbed at the media: not only that it's liberal, but that it supports the establishment (or "corporations").

Around the same time as Cronkite took his stand, American media broke a number of big stories that hurt Republicans more than Democrats: the Pentagon Papers, Watergate, Vietnam in general. Not only did these stories undercut Republican arguments, they were pursued in a way that flouted authority: refusing to name anonymous sources, reporting classified information, and in the minds of some conservatives, endangering American troops. On top of this, reporters beginning in this time exposed conservative Americans over and over again to stories on poverty, race relations and other subjects that those conservatives didn't see in their daily lives; it looked to the conservatives like reporters were highlighting liberal pet causes. And the idea of a "Silent Majority" that cared about different things than what the media reported on was to some extent validated by Republican electoral wins. Conservatives' perception that the media is liberal, then, was probably fairly widespread by the 1970s. It was then seized on by Fox, by Rush Limbaugh and by others who marketed, very loudly, that mainstream media was biased against conservatives. Without any equally loud voice arguing the opposite (although Noam Chomsky and others certainly did make the argument), the conventional wisdom of a liberal media was well established by the 1980s.

Edit: Google ngram results show that book results for "liberal media" start in the late 1960s, rises through 1990, then really explodes starting right when your personal recollection starts.

To follow up on what OP is asking: does this come from the term "liberal media" which has more to do with media being free and not them having a political association like the term "liberal" currently has? (Think of "liberal democracy" too)