Well, that's an enormously far-reaching question, and I guess that the amount of literature written on that subject could easily fill a kilometer or two of bookshelves. That said here is my attempt at a short and brutally oversimplifying summary!

The immediate changes for the Catholic Church were twofold: Firstly, they lost a lot of property and influence in the areas of Europe that turned Protestant. There was a vast number of monasteries dissolved in e.g. Northern Germany, whose property in most cases was confiscated by the rulers. So that's a lot of money down the drain, for one thing. But even more important than the simple loss of worldly possessions and political clout was the fact that suddenly, there was an alternative to Catholic dogma.

There is a good argument to be made that you cannot really speak about the "Catholic Church" prior to 1517. There were a good number of heretical movements during the Middle Ages, yes, but for the most part they failed to receive lasting support by the elites and got stamped out over time. The Orthodox Church to the East didn't really try and encroach on Catholic turf, especially after the devastating loss of Constantinople in 1453, and it was/is pretty similar to Catholic dogma and liturgy all things considered. So again: Medieval Europeans had no real alternatives to the Catholic Church, which for them was indeed simply the Church. That change in perception - an alternative Christianity is not only possible, but it exists right there - had tremendous repercussions for how people perceived themselves and the world around them, and it would continue to reverberate through the next centuries of European and Church history.

Turning from the Church into the Catholic Church meant it was now necessary to meaninfully differentiate yourself from the upstart Protestants. This process is called "confessionalisation" (after the German historians Wolfgang Reinhard and Heinz Schilling) and entails things like the standardisation of theological dogmas and liturgical norms, the spread of apologetics and religious propaganda, the establishment of those new rules in the general populace by education and propaganda and so on. This process also happened in the other two big denominations that sprang up in Europe with the Reformation, i.e. the Lutherans and the various Reformed Churches.

For the Catholic Church, this process is often called "Counter-Reformation", a term that has been wildly and bitterly argued and where entire generations of historians have poked holes into, but still has managed to stick around. The most important point in the beginning or establishing of this "Counter-Reformation" is generally said to be the Council of Trent, an assembly of hundreds of bishops, prelates, theologians and ambassadors (some of which even representing Protestant territories, funnily enough) lasting from 1545 to 1563. The Council formulated a long list of Catholic dogmas, standardised Catholic liturgy and (translated into modern manager speak) formulated an "action plan" in their endeavour to stem back the tide of Reformation. Some of the highlights decided on by the Council include the abolition of the sale of indulgences, the establishment of a standardised education of priests in the form of seminaries, defining the dogma of transsubstantiation, strengthening the role of bishops within the ecclesiastical hierarchy, proscribing the exact beliefs and religious practices to the laity (you could also call this a turn from religion-as-practiced to religion-as-prescribed) and many more. The changes, both officially promulgated and de facto effected, are honestly too many to count. Suffice to say that the Catholic Church of the Early Modern Era is very different from the Medieval Church.

Now, I need to stress again that all of the above is a gross oversimplification, and I left out many aspects that ran counter to the overarching narrative of Reformation and Counter-Reformation. Just to name a few: Catholic and Lutheran liturgy needed decades, if not centuries, to meaningfully diverge; many changes prescribed by the Council of Trent were only implemented centuries later; more recent scholarship has found that in Germany confessional identity only really came about during the 19th century, at least outside of religiously mixed areas; the turn to religion-as-prescribed was also more of a 19th century thing; the Catholic Baroque actually rediscovered many practices and ideas that used to be popular during the Middle Ages and then were almost during the Reformation and so on.