In a word, artillery. I'll base this answer off what I've seen in Ireland, the exact process might be different in other places.

The introduction of gunpowder prompted a switch from castles to "trace italienne" style fortresses. The vertical walls and battlements that characterize medieval castles were a substantial deterrent to assaulting infantry, but they were weak to bombardment. Once a breach was created, infantrymen still had to take it (not a fun job, the first group of attackers were sometimes called the forlorn hope) but the castle was likely to eventually fall.

You then see a transitionary period, where traditional castle architecture was adjusted to incorporate things like gun loops and artillery platforms. For a good example, see Castle Cahir.

As field artillery grew more powerful, this solution was no longer adequate. This is when you see the development of the full blown trace italienne style fort. For an early example see James's Fort in Kinsale, Ireland. In the same town, Charles Fort is a later upgrade of the same basic idea. The star shape produced by the gun platforms means that the guns have overlapping fields of fire. The walls no longer have to be exceedingly high, but they need to be thick and absorbent enough to stand up to prolonged siege bombardment. This is why you see earth incorporated more into fortification, it absorbs shock better than rock.

Cosmetic forms of castle architecture continue to be used, but they're now more about status than actual fortification. Even as the military moved to new styles, the popular association of "big stone building=rich guy" seems to have stuck around. There's some disagreement about the function of "tower houses" in the British Isles, but I would argue they fit this function.

For a good overview of fortification developments in the early modern period, see Siege Warfare: The Fortress in the Early Modern World 1494-1660 by Christopher Duffy.