There's no single unifying reason. At different times and in different places, many cities found city walls were obsolete for a number of different reasons.

Although I only know some trivia here and there about static urban defenses in european new world colonies, I can instead speak to Italian urban history and urban planning, especially in Italy's economic capital: Milan. In Italy, traditional wall construction fell onto the back-burner of military engineering over the course of the 16th century. Over the course of the Italian Wars that century, increasingly large field battles made sieges a less prevalent feature of warfare, and when sieges did happen artillery did a lot to shorten them. When cities and towns chose to invest in new fortifications, complex earthworks taking into consideration artillery technology were much preferred over upright walls of stone and brick. These earthworks, often star-shaped, came to be known in the rest of Europe as the "Italian Outline" of fortifications, also contemporarily referred to as the "Modern Outline." Concurrently, many cities found that thanks to modern artillery and increasingly large field armies, they only really needed to fortify strategic choke points: you can see the places for artillery batteries and star-shaped forts in this diagram of the sixteenth-century fortifications of Florence.

However, this doesn’t mean the construction of circumferential urban fortifications abruptly stopped altogether. In Milan, low-lying sloping stone fortifications built with artillery in mind were constructed in the second half of the 16th century. Smaller towns with more traditional fortifications didn't do away with their walls just yet either: walls continued to be a convenient way to collect tolls from people moving goods in and out of urban spaces. In times of war, old walls would still keep out plundering bands of soldiers, while in times of peace the same walls would keep brigands at bay. Walls were convenient markers for the city limits, and in less dense towns walls would also keep livestock from wandering off at night. For this reason, even small towns could be expected to be surrounded by a palisade of stakes or at the very least some sort of fence, even if the immediate military value of these barriers was negligible.

The ultimate obsolescence, and actual removal, of urban walls only occurred in the eighteenth century. In 1750, the governor of Milan, Giovanni Luca Pallavicini-Centurione, decided to transform the entirety of the city walls into a public park. They became a series of hanging gardens of sorts, and the ramparts were worked into a series of elevated promenades. Most Italian cities initially mirrored this approach, and the city of Lucca in Tuscany is perhaps the best preserved example. However, the realities of urban density and the boom in population growth during the industrial revolution made conservation increasingly impractical: in this map, you can see how as late as 1832 urban density never really pressed up against the outer ring of walls of Milan. However, by 1860 urbanization had progressed right up against the walls, while by 1910 you can see how dense urban construction was affirmed outside the city walls (you can even make out a streetcar on the left side of that last photograph, running its route outside the walls). Although the Milanese walls were a quaint historic artifact and a favorite recreation space (Stendhall would write that on a sunny day, walking along the walls you could clearly make out the white peaks of the Alps set against the city's skyline) the way they sliced up the city was very impractical. As the pace of life quickened and goods and services consumed by people grew, the traffic of carriages, streetcars, and later trains and automobiles, all funneled through the city gates eventually led the walls to be perceived as a barrier to economic growth and incompatible with modern lifestyles.

To their credit, the authorities in Milan initially did their best to retain the city walls, opening up new entryways with vehicular traffic in mind. The city's first railway station was even placed immediately outside an existing gate. However, with the growth of railway travel, new stations handling commuter and long-distance travelers needed to be built; and although initially new gates in the walls were constructed, the need to place a railway station inside the city center (the modern Cadorna Station) made the contiguity of the walls impossible to retain. By the late 19th century, urban growth had led city authorities to commission a detailed growth plan which called for the demolition of the walls in favor of wide circumferential boulevards.

The demolition of the walls of Milan was never total; some gates and tollbooths are still standing, while a few tracts of the walls are also recognizably intact (a tract is even part of an exclusive spa). While it's certainly a decision that makes history buffs and architectural conservatives cringe, the retention of the city walls in the industrial age would have been very impractical. Across Italy and indeed Europe, very similar considerations were made. Some were systemic, as the Milanese were, others instead underwent a gradual disrepair and eventual demolition, as in Italy's other largest metropolitan area: Rome, where the walls are in many places still standing in varying degrees of disrepair and conservation across their circumference. In the city of Bologna, in the early 20th century city authorities even encouraged residents to help tear down the city walls, such was the sentiment they stifled the city's walk towards modernity (history would have the last laugh though: in the post-war era the city council in Bologna implemented one of the strictest building conservation codes in Italy).

It's common to retain some architectural features of ancient walls here and there in European cities, even where they have been largely demolished. Because they are often ornate, gatehouses are a favorite feature to retain. Many ancient cities which lost prominence to more industrial neighbors also retain intact defensive walls, or never felt the need to modify them beyond their transformation into public spaces like parks and gardens. Some small towns never even grew beyond their walls. In sum, the conservation of urban defenses is in no way uniform and is often subject to a variety of local considerations. However, it's fair to say that the largest cities very often went down the path Milan did, and felt the need to do away with their walls.