This is not quite a historical question, as languages are not categorized into language families based on historical relation. Instead, they're categorized by which common ancestors they have - much like how we categorize animals and plants with biological taxonomy. In this case, two languages are in the "same language family" if they both split off from the same language some time in the past. It's usually fairly clear-cut which family a language belongs to, even when it has been strongly influenced by other languages from different families. Linguists have gotten quite good at reconstructing proto-languages (that is, early common ancestors) from extant languages.

Khmer and Vietnamese can both be traced from the Proto-Austroasiatic language, so they're both in the protoasiatic language family - though eventually, that language split and changed and split again until they became Khmer and Vietnamese (and some other languages in the same family). For comparison, most Northern Indian languages are from the Indo-European language family. The Proto-Indo-European language is the common ancestor of a lot of languages that you might not think are related, such as English, Russian, Hindi, Latin and Farsi.

Hope this answer is allowed within the rules of the subreddit, it was a bit unclear to me.