In short, the French did not (at that time) try to take the duchy by force, because William was still a vassal of the French king in his capacity as duke of Normandy. It’s important to understand the distinction between personal and real union as it existed in western European feudalism- while any number of titles might be concentrated in the person of a single ruler, that did not necessarily mean that those lands were combined into a single polity. One can find numerous other example of this in English history; for one, after James VI of Scotland acceded to the English throne as James I, the two countries maintained separate governments for nearly a century, despite having the same head of state. After George I, elector of Hanover, became king in 1714, the duchy of Brunswick-Lüneburg (later the Kingdom of Hanover) remained a possession of the British kings until Victoria acceded to the British throne. Hanoverian succession law did not allow for a woman to take precedence over any male relative, so the title passed to her uncle, Ernest Augustus. In the same way, Normandy and England did not become a unified state, and there was no reason they would remain under the same holders forever.

Indeed, after William the Conqueror died, Normandy and England went to separate sons, Robert and William, and the matter might have ended there. However, after William II of England died, the third son, Henry, seized the English throne. Robert, duke of Normandy, invaded to press what he saw as his superior claim to the throne; he was defeated, and a few years later Henry struck back and took Normandy. He installed his only son and heir, William, as duke, indicating an intent to maintain the duchy as a possession for future generations.

Now, this is where it gets tricky. Normandy and England were still de jure separate fiefs, but Louis VI of France was understandably not excited at the prospect of having to deal with a powerful and potentially hostile vassal who held a separate kingdom. (This consternation would prove to be warranted [see: the next 350 years of Anglo-French relations.]) Louis supported the claim of Robert’s son to Normandy and invaded in order to press this claim. Henry I won, retained Normandy, and the stage was set for several centuries of intermittent conflict, culminating in the English kings contesting the throne of France itself.