Where the political elite that conquered a country are from doesn't necessarily impact the language. French isn't considered "Latin based" due to the history of the land that it is spoken in, it is simply a linguistic descendent of Latin in fact. Its vocabulary, syntax, etc. are all for the most part Latin, not Germanic, based.

When the Franks took over the area of France they didn't genocide the people living there. They were a ruling elite and adopted many aspects of the Local, Roman derived, culture (ex. Being crowned "Holy Roman Emperor", adopting the Latin title Carolus Magnus)

The same is true if you look at England, even though it was conquered by French (Romance) speaking Normans, the English language remains a Germanic language, though there is a huge impact vocabulary wise from French.