Hakka, Cantonese, Mandarin, Wu, and a few others, are the descendants of Middle Chinese. Middle Chinese and Min are descendants of Old Chinese. Min was an earlier branch than those others, and thus we find Southern Min (Taiwanese, Teochew, Hokkien among other names/groups), Nothern Min, Eastern Min, Puxian Min being really quite distinct, in some regards as distinct as Cantonese is from Wu (Shanghai area).

There's some argument about what I just wrote, but it's basically the common view and detractors are a small minority.

As to what languages were there before, one of the answers is "just about every Southeast Asian language you've heard of". So Thai/Lao and Zhuang were one. The modern Dai 傣族 ethnicity are the members of this larger old group that remain in China. The Shan in Burma, the Ahoms and Phakes and Khamtis in India, and the Dais in China are all branches off a Tai-speaking group centred in modern day Yunnan and into Guangxi. So Thai & Zhuang were one such group. Thai, Lao and Zhuang belong to the Tai-Kadai language family.

The Miao 苗族 also known as the Hmong are another group. The Hmong who today are found in Southeast Asia (and further abroad) had ancestors who were originally living further north in what's now China.

There are other groups as well. The She 畲族 are a prime example of one such group which had an impact. The She were living (and many still do) in Zhejiang province, Fujian, and throughout Southeast China. Much of the substrata details in Hakka are pointed to the She as the source. The She spoke a Mienic/Yao language, which is closely related to Hmong/Miao. You'll see "Hmong-Mien" as a language family outside China and in Mandarin the same group is called "Miao-Yao".

There's a 2004 paper by Laurent Sagart, Gan, Hakka and the Formation of Chinese Dialects, which is available online and is a good overview of this sort of substratum development. It pretty directly answers your question, in linguistic terms.

There we also Austronesian speakers in the area, though it's hard to say how many and when. Off the top of my head I don't have much to offer on that point.

There's a huge amount written on this topic, mostly in Mandarin. I don't have the papers handy but if you can read Mandarin I can suggest some authors to look into. The Sagart paper gives some examples of reflexes of these other languages, but it's more in the form of phonology than lexicon.

---

edit: some further reading as requested – 

Luō Měizhēn 罗美珍 has written a fair bit, e.g. 《畲族所说的客家话》 from 1980 and 《从语言上看傣、泰、壮的族源和迁徙问题》。 Also Wú Zhōngjié 吳中杰 《畲話和客家話》 and also Dèng, Xiǎohuá 邓晓华《客家话跟苗瑶壮侗语的关系问题》。 Then also 《壮话与白话、客家话、闽话的共同特征》 by Wéi Dá 韦达 for another.