In the United States, the idea of a "think tank" broadly evolved out of several arrangements in the immediate post-WWII period. Project RAND was started in 1946 by the US Army Air Forces to provide independent analysis of military policy. It became an independent, non-profit research and development corporation in 1948, and became something of a template for many other similar organizations. It was entirely dependent on government contracts for its work, and employed a wide variety of interdisciplinary scientists and thinkers aiming towards (largely quantitative) analysis of problems the government and military had. Specifically it adapted the wartime Operations Research methodology into what they called Systems Engineering methodology, a quantitative method for making choices about finding optimal arrangements of systems and strategy, largely for military questions. They also did important work with the applications of game theory.

Think tanks were essentially private institutions that employed academics. A separate but parallel development was the so-called Summer Study, which were temporary groupings of academic consultants on government problems. These got started in the early 1950s, with efforts like Project Charles and Project Vista. Some of them got made into permanent (albeit rotating) institutions, like the JASON group.

The term "think tank," used in this sense, was first used around 1958, according to the OED. But these types of organizations existed well before that. Google Ngrams suggests the term really took off in the 1960s, which makes sense, given how prominent these kinds of institutions became then, during the "whiz-kid" era of governance. I think it may be historically difficult to disentangle the idea of a "think tank" from that of a lobby, but RAND makes a good candidate as the first real "think tank," in terms of its methodology, patronage, participation, lack of being a real "lobby" of sorts (though they were so dependent on the US Air Force that their recommendations were almost all pro-USAF), and so on.

It's not a scholarly treatment, but this post by Adam Curtis gives some of the history of think tanks in the UK (it'll work better if you're in the UK or can spoof your location to watch the video content).