They gave up at the treaty of Paris in 1763. This treaty was supposed to send the seven years war, or as Americans more "colorfully" call it, the French and Indian war. This was the same conflict in which the British Army lead by general Wolfe took Quebec, so the British were occupying it at the time of negotiation, but a big part of the negotiations consisted in agreeing about what belligerent would keep what territory. They concluded that the French gave up their claim to Canada in exchange for getting uncontested control of Guadeloupe, a sugar-growing island. It was much more profitable for the French, whereas Canada's resources in fur and land were more difficult to have use of. They were literally losing money by keeping Quebec, so trading it off for concessions elsewhere seemed strategic. This marked the end of French interest, because they didn't want an unprofitable colony: to them Quebec was, as Voltaire put it, "quelques arpents de neige" ("a few acres of snow").

Some French Canadians expressed abandonment over this, but the English rule was accepted mostly peacefully. Quebec nationalism in terms of taking a separate nation state is a more modern idea. Instead, French Canadians embraced "la survivance" - the idea that they should take any path to maintain their unique culture and language, whether that means co-operation with English-speaking authorities or some kind of resistance. The Catholic Church, which was important as leadership to Quebec, made deals with the English and helped them settle in. The Anglophones of the time commented on this as "stubbornness" or "natural accommodation to authority", especially governor of Quebec Guy Carleton, whose reports to the English government encouraged the adoption of the Quebec Act. The Act gave Quebecers the legal right to be Catholic, keep the Civil Law, and speak French in public affairs. It also pissed off the 13 colonies.

"But wait," you may be saying, the French allied with the American Revolutionaries. Did the French try to get Canada back through that method? The Americans even invaded Quebec." The answer is a bit anticlimactic, because they didn't. France sought more of a strategic weakening of their longtime English enemy, but French officials didn't cite getting Quebec back as a motivation so much as getting even and supporting the Americans who shared many ideological commitments. France opted to send support to the rebels rather than to go into a full war with Britain. The Americans were the ones who thought they might take over Quebec with help from the poor Quebecers who were just waiting to throw off the English, right? Wrong.

Over the years, French Canada developed a distinct identity that became more different as the French went through the Revolution and the Canadiens remained a deeply Catholic culture. Relatively quickly, France became so different that French Canadians didn't really have a lost mother country to want to reunite with, but rather a former metropole that had grown into a very different place. The colony had been pretty minor in the French geopolitical strategy and, as we mentioned, a money sink, so no political will existed on that end either.

Edited: because British isn't the same as English! Sorry guys.