Cassell's Dictionary of Slang dates it from 1961, while The Concise New Partridge Dictionary of Slang and Unconventional English dates it from 1946. I'm inclined to go with Partridge on this one, largely because of something William Safire wrote in his Political Dictionary:

"Baseball's left field was the place of far-outedness by the late 1950s. The phrase out in left field came to have two meanings: 'removed from the ordinary; unconventional' and 'out of contact with reality, out of touch.' A 1960 New York Times report on the state of U.S. scientific research used the phrase in the first sense ... The second meaning is illustrated in a 1959 Time magazine article ... Sports fans differ on the reasoning behind the metaphor. Some suggest that left fielders play farther out to get the power balls hit by right-handed players. Others note that in most of the other, asymmetrical ballparks — Yankee Stadium, Ebbets Field, Forbes Field among others — left field was deeper than right field. That gave left-handed batters an advantage because distances to right field fences were shorter."

I've bolded what I think is the correct answer to your question.

The first use of "left field" in a non-baseball context within the New York Times archives is this 1946 article dealing with a shipping squabble.