They knew how the U.S. pulled it off because they saw the ships and saw the bombers.
A raid by carrier-borne aircraft on the Japanese Home Islands wasn't unexpected. Admiral Matome Ugaki, chief of staff of the Combined Fleet, referred to the possibility as early as Feb. 2, 1942. To safeguard against a strike, a line of picket boats were stationed between 650 and 800 miles offshore. This was unknown to the Americans, who were surprised when they detected the pickets on radar as they approached the raid's launching point.
The Japanese supposed that carriers would have to approach within 200 miles of their targets in order to launch a strike. The Americans, as surprised as they were by the Japanese pickets, had a surprise of their own. The Japanese didn't expect that the U.S. would fly long-range land-based bombers off their carriers.
The Americans had intended to launch 400 miles off the Japanese coast, but the picket boats forced them to do so almost half again that distance. As a result, the Doolittle raiders would be forced to ditch, rather than land, in China.
Despite warnings radioed by the converted whaler Nitto Maru No. 23 (its radio messages were overheard by the American task force), Japanese defenses were caught almost completely by surprise. Not a single B-25 was brought down by antiaircraft fire or fighter aircraft.
The physical damage inflicted by the Doolittle raiders was minor, but the psychological impact was huge. "This shouldn't happen. This just should not happen," muttered Admiral Osami Nagano, chief of the IJN's general staff, after he heard word of the bombing. Admiral Yamamoto claimed he was ill and stayed in his cabin for an entire day.
The Japanese scrambled to catch the raiders -- the carriers Akagi, Soryu, and Hiryu, returning to Japan after a raid on Sri Lanka, were sent after the Americans to no avail.
The raid changed the Japanese Army's thoughts to action in the Pacific. General Shizuichi Tanaka, commander of the Eastern District Army, had been opposed to operations against Midway Island before the Doolittle raid. After the raid, he reversed course and pledged his support. Even more interesting, on May 25 (days before the Midway attack launched from Japan), the Army issued orders for units to begin preparing for an amphibious assault on Hawaii, their training to be complete no later than the end of September.
On the American side, the Doolittle raid meant the aircraft carriers Hornet and Enterprise were not available for the Battle of the Coral Sea in the south Pacific. Had the Americans had four carriers at that first carrier-to-carrier battle, it's interesting to imagine what could have resulted.
Did the raid cause Japan to station more fighters in the home islands for defense against another such raid at the expense of more fighters at the front line where they would have been of more use?
Summary of the Doolittle raid:
https://www.youtube.com/watch?v=1w30FkSXyTE#t=8m53s
Japan's reaction:
https://www.youtube.com/watch?v=1w30FkSXyTE#t=10m15s
EDIT:
So, I'm being downvoted here, presumably for providing a video source and presumably by people who did not have the patience to watch said source.
I say that, because the brief ~3 minute segment provides a better overview of the Doolittle Raid than /u/The_Alaskan's comment, and it gives a much better description of Japan's reaction (/u/The_Alaskan gave an incomplete explanation for the reasons behind the physiological reaction).