So, I will start off by admitting that I know more about the Vatican's intelligence apparatus than I do of their explicit connection with the Ustase, but I think I can still answer this for you.
There is no doubt that the Vatican was aware of what was occurring in Croatia during the war. None. The Vatican's intelligence apparatus, known as "The Entity", operates incredibly quietly and incredibly efficiently. It was operating in full force during the Second World War, providing the Vatican with an up to date picture of situations occurring across the globe. This almost certainly included Croatia, as it was a large Catholic population with many priests present, and it was through the reports of priests that much of The Entity's intelligence arose. According to Michael Phayer's book Pius XII, The Holocaust, and The Cold War, the Pope was aware that atrocities were taking place and that Catholic priests were even taking part in them. Now, it's possible that the Vatican may have been disregarding reports about these atrocities as propaganda, but as the reports continued to come in, it seems far more likely any such denials were more a matter of political/diplomatic cover than actual ignorance. They may also have handwaved these crimes as the birth pangs of newly independent state, but again, that rings hollow considering the power of The Entity and the presence of so many Catholic officials on the ground in Croatia. Essentially, it would've been impossible for the Vatican not to know about what was occurring considering just how much death was happening. 500,000+ Serbs, Gypsies, and Jews (the conservative estimate, mind) do not just die without anyone noticing. Finally adding to this certainty that the Vatican had knowledge of these atrocities is the fact that they definitely heard about them from the Archbishop of Zagreb, Aloysius Stepanic, who, though initially in favor of the Ustase regime, turned against it over the course of the war to the point where he felt the need to speak out publicly about what was occurring (took him a while though).
So, we know that, without a doubt, the Vatican was aware of the crimes happening under the Ustase regime. However, were they complicit? My answer is no, they were not. The Vatican was certainly guilty of failing to condemn what was occurring in Croatia. As John Cornwell notes in an article about Pius XII:
Despite a close relationship between the Ustashe regime and the Catholic bishops, and a constant flow of information about the massacres, Pacelli [Pius XII] said and did nothing.
Pius XII granted an audience to the leader of the Ustase Ante Pavelić, though it's worth noting he was only granted a "devotional" audience, not a diplomatic one. Still, Pius XII spoke with him and did not, as far as we know, condemn him for the killings that were occurring. Evidently, Pius XII was more than ok with turning a blind eye towards the massacres that were occurring, or, at the very most, was willing to do little more than send the occasional diplomatic communication saying that the Ustase should tone it down a bit. According to Phayer, the Pope (and the Vatican at large) felt it was better for Stepanic to try and rein in the Ustase along with gentle diplomatic pressure rather than outright denouncing them for what was occurring. Cornwell attributes this to the Vatican's desire to expand Catholicism into the Balkans and its relative unwillingness to derail a potential ally in that effort.
On the ground in Croatia, Phayer notes that local Catholic priests were not opposed to taking part in the bloodshed that was occurring. However, we also know that some priests and high-ranking figures, including (eventually) the Archbishop Stepanic and the Apostolic representative to the Ustase regime, Giuseppe Marcone, actively worked to try and counteract or lessen killings that were occurring. There is no evidence that participation in the crimes of the regime was by any means a blanket policy among Catholic priests in the area or even a "common" behavior.
So, back to the question of complicity. I would argue that the above does not constitute complicity on the Vatican's part. They are no more complicit in the Ustase's crimes then say, the United States was complicit in what occurred in Rwanda. Much like the Vatican, the US was well aware of what was occurring in Rwanda during the genocide, but it failed to act effectively to try and prevent what was occurring. This does not make the US complicit in genocide. It makes them callous and irresponsible, but not complicit. The same can be said of the Vatican and the Ustase. The Vatican did not participate wholesale in what was occurring. They did not endorse it (though they were in favor of mass conversions, they did not mean conversions at the barrel of a gun). As an institution, the Vatican even weakly condemned what was happening through its half-hearted diplomatic appeals (while at the same time offering Pavelić an audience, so not really backing up what they were saying). But truly, Pius XII did pretty much nothing about massacres he was completely aware of when he most likely could have done something. That does not make him complicit. That makes him an irresponsible, morally and diplomatically weak leader.
Sources
The Entity: Five Centuries of Secret Vatican Espionage by Eric Frattini
Pius XII, the Holocaust, and the Cold War by Michael Phayer
Hitler's Pope by John Cornwell https://www.vanityfair.com/style/1999/10/pope-pius-xii-199910