A professor I had once casually mentioned that Canada was almost taken over by just a few thousand Irish people, but that they gave up because they didn’t think it was worth it. Is this anything? Is this remotely close to something that happened?

There's always more to be said, but in the meantime you can find some answers in this post about the Fenian Brotherhood by u/enygma9753.

The Fenian Brotherhood was a secret political organization dedicated to the cause of Irish independence. They were active in Canada, the U.S. and Britain during the 19th and early 20th centuries. Their intent was to hold Canada -- still a British colony then -- hostage until Britain granted Ireland its independence. They also had planned to instigate a general uprising within Ireland.

As many of them were recent veterans of the U.S. Civil War, they had initial success against inexperienced militia at the Battle of Ridgeway in 1866 near Fort Erie Ontario, across the river from Buffalo NY. Their force was small (under 800) and was in no position to effectively conquer Canada, as the U.S. government itself cut their supply lines and did not sanction their activities. President Grant even had ringleaders arrested after subsequent raids on the St. Lawrence and Vermont frontiers. These all ultimately failed, repulsed by Canadian militia and the intervention of American authorities. The attacks demonstrated the need for common defence in Canada and helped to hastened the union of Canadian colonies into one nation in 1867.

Britain was still responsible for the defence of Canada at the time and had real concerns about raids and terrorism in the new Dominion of Canada. (Fenians had been actively organizing across the border in Washington and Oregon.) When the Canadian Pacific Railway was completed in 1885, Britain sent warships to the harbour just off the railhead and its docks. A large body of troops were also sent west by rail to the Lower Mainland of British Columbia on the first train.

It would be more accurate to say the Fenians were compelled to give up their efforts, due to weak public support, poor organization and frequent in-fighting. The prospect of capturing Canada was negligible, when the U.S. government opposed and disrupted their activities and Britain was still prepared to defend its imperial interests overseas.

Canadian colonies had legitimate fears that Britain may not be as invested in their fate someday, which ushered in their eventual Confederation.