You're correct that the genesis can more or less be traced back to World War II. The US Army had some experience with various "elite" infantry units before that, but they don't share much organizational DNA with the special operations forces that arose during World War II and after.

Perhaps it's appropriate that the origin story of Army Special Forces, who consider themselves "quiet professionals," isn't well known. However, the "special forces" units of World War II, and especially the Army units, such as Merrill's Marauders, the First Special Service Force (also known as the "Devil's Brigade") had provided a foundation for doctrine and training. The Office of Strategic Services (OSS, popularly remembered as the forerunner of the CIA) played a role in arming and training foreign fighters, which required language skills and foreign intelligence that hadn't been required before the war. The OSS also conducted what you'd classify today as "direct action" missions during the war.

Robert A. McClure (there was also another Army officer about the same age/rank named Robert B. McClure) was an officer on Dwight Eisenhower's staff in Europe in charge of the Information and Censorship Section (INC), which included the Psychological Warfare Branch. Later, he became Chief of the Publicity and Psychological Warfare Division. Among other things, this gave him experience with the Office of War Information (OWI, basically Allied propaganda) and the OSS. Most of the Army's psychological warfare apparatus was not preserved after the war ended, but McClure hung around. When the Korean War began, the Army re-established the Office of Psychological Warfare with McClure in charge.

OPW included a Psywar branch but also a Special Operations branch. McClure and his people set up shop in Kansas initially, but the Psychological Warfare Center was established at Fort Bragg in 1953 (and is still the home of Army Special Forces today). Even before moving to Bragg, however, the new unit was stocked with some of the veteran commanders who had experience in special operations from World War II. Col. Aaron Bank, who had participated in OSS operations in Europe and the Pacific, was one of them and was the unit's first commander. Col. Wendell Fertig and Lt. Col. Russell Volckmann, who had both been in the Philippines when war broke out and fought as guerillas after the American surrender there, also joined the unit. At least a couple others who fought with Merrill's Marauders were also early additions. But don't discount McClure's importance to all this or his credentials. After the Psychological Warfare Center was up and running, he was sent as a military attache to Iran in March 1953. Later that year, a coup deposed the president who had threatened to nationalize the oil industry and re-installed the shah, who was friendlier to the west.

There was concern in the early days about whether the new Special Operations Forces would have an overlapping mission with the Rangers, who already existed and were fighting in Korea. Ultimately it was decided that the new unit would focus on "subversive activities," working with indigenous populations, and guerilla warfare. By 1956 the Special Operations Forces had grown large enough that the Psychological Warfare Center was rechristened the Special Warfare Center and School.

SOURCES

"Oppose Any Foe: The Rise of America's Special Operations Forces" by Mark Moyar, 2017

"Major General Robert A. McClure" by Jared M. Tracy, from Veritas, Vol. 7, No. 2, 2011. https://arsof-history.org/articles/v7n2_mcclure_page_1.html

"US Army Special Warfare: Its Origins - Psychological and Unconventional Warfare, 1941-1952" by Alfred H. Paddock, 1982