A decent answer has already been linked that covers a lot of relevant points, but there's a little more to say about the particular cultural dynamics of the way that cricket and other sports develop (or don't) into 'modern' (ie commercialised, professionalised, mass spectator/participant) activities. These processes were very fluid in the late nineteenth/early twentieth centuries, and could lead to heavily diverging outcomes across contexts. My interest here is drawn partly by similar trajectories in Scotland - where one can definitely point to climate-related reasons, but a more complex story surrounding identity and purpose can also be told.
Cricket, like other British imperial games, did indeed enjoy some initial popularity and public profile across much of Canada before and after Confederation, as outlined by the linked answer. This wasn't just a matter of international teams touring Canada (and vice versa), but also relatively dynamic local club scenes and leagues, which often enjoyed a relatively positive public image compared to emerging sports such as hockey, which was initially seen as disruptive and uncouth, not least because it was largely being played by working-class youth. cricket, on the other hand, was leant social respectability by its association with Englishness - not only did the numerous first-generation migrants from England want to keep playing the game, for those Canadians who valued the cultural cachet that came with proximity to Englishness, cricket was a useful signifier of their civilised imperial tastes and affiliations.
In contrast to somewhere like Australia, where playing cricket was a social sphere in which class boundaries were broken down, Canadian cricket was therefore poorly culturally situated to become a 'modern' mass sport in the twentieth century. If much of the point for many players was to demonstrate belonging to an amateur social elite, then not only would attracting a wider player base become difficult, it would defeat the some of the purpose of playing. As such, amateurism remained entrenched and the player base stagnant or declining, with extremely limited prospects for commercial exploitation. This was compounded in turn by the decreasing proportion of English migrants, as well as the greater and greater cultural influence exerted by the more proximate United States (with baseball as much as hockey representing a clear competitor for space and players). As Canadian national identity grew in importance relative to Canadian imperial identity over the course of the twentieth century, the social impetus to play also lessened (though didn't disappear completely).
That said, this is very much the story of white Canadian cricket. Just as the game was originally popular due to Canada's English diaspora, so too did cricket clubs serve as an important social, cultural, economic and sporting reference point for the increasingly large Carribbean and South Asian diasporas. In this sense, cricket is hardly an unpopular sport in terms of ongoing participation, but in a way that is very different either from our conception of a 'modern' sport grounded in a national cultural hegemony, or the original social and racial basis for cricket in Canada.
Sources:
There's a good overview of the historiography of Canadian cricketing, as well as an introduction to some key cultural perspectives on sport, in John G. Reid and Robert Reid, 'Diffusion and Discursive Stabilization: Sports Historiography and the Contrasting Fortunes of Cricket and Ice Hockey in Canada's Maritime Provinces, 1869-1914', Journal of Sport History 42:1 (2015), pp. 87-113. On the last point, see Janelle Joseph, Sport in the Black Atlantic: Cricket, Canada and the Caribbean diaspora (2017).
Here's an answer to a similar question by u/McFoodBot "Most of the best national rugby and cricket teams in the world are from countries which were part of the British Empire. Why have these two sports (seemingly) failed to gain as much traction in Canada?". Hope this helps!