I've heard from multiple sources (e.g. friends' parents, my own parents) that the Japanese people were originally prisoners sent to the island from China. Obviously, differences in the languages and other things throw a whole lot of doubt into this. But, at any point in ancient or imperial Chinese history has there been any record of Chinese prisoners being sent to Japan?
This, for all intents and purposes, is BS told to you from your sources, for the simple reason that there exists no record that stated China sent prisoners to Japan.
There is a debate*, however, about whether or not the Japanese imperial family is actually descended from Xu Fu (Jofuku in Japanese), who was sent by Qin Shihuang to find the mythical the island of Penglai to search for the elixir of life. Most historians believe that Xu Fu found Japan and told Qin Shihuang that he had reached Penglai. On his second expedition, he brought with him 3,000 men and women and never returned. The Japanese believe Xu Fu settled in Japan (as evidenced from this Japanese tourism bureau page) and there are shrines and monuments to him in Japan. But whether or not he actually founded the imperial family or what his impact was to the development of Japan is unknown.
(*) This theory is advanced by a tiny minority of Japanese historians, but it has been picked up by a lot of Chinese nationalists.