It's not so much of an either-or as much as a both-and. On paper, the church was opposed to it for exactly the reasons you state---but at the same time, the Catholic Church was also one of the primary employers of castrati, even in the Sistine Chapel Choir.

Castrati actually endured much longer in Church than in secular contexts---in 1813, Rossini, an opera composer, believed he was writing for "the last" castrato (Feldman 260), and at that time they were already obsolete. Mozart (for example) had written numerous operas with tenors in leading roles decades prior. Yet there would be castrati in the Sistine Chapel for a full century further; Alessandro Moreschi sung there until 1913.

Some might even argue that the Church encouraged it culturally, insofar as it was considered a sort of blood sacrifice. Martha Feldman puts it nicely:

Always mediating the phenomenon were Catholic religious ideas, often intermixed with rural folk beliefs as well as familial strategies for distributing wealth and functions within a system of primogeniture. Legally the church condemned the practice as being against the order of nature [...]. And yet proscriptions do not map onto the symbolic load castration bore. In some sense castration for singing, as a sacrificial offering to the church, was much like joining the priesthood. (7)

The church also opposed castrati insofar as it forbid them from marrying (more on that in this old post on this subreddit by the fittingly-named /u/caffarelli)---although, in a sense, that renders them no different from priests.

Work cited: Feldman, Martha. The Castrato: Reflections on Natures and Kinds. University of California Press, 2016.