I cannot answer your main question. My language skills are insufficient to research this particular issue. I can shed some light on a part of your final question, particularly this portion:

what was the role of the Vatican in this?

There is a real dearth of good English-language sources on the issue of the Ustase and the Vatican. Most of the books that even mention the topic feature only brief mentions in a larger discourse, usually regarding Pius XII. Most of the books on that man are highly flawed. In far too many cases the bias of the author coming in to the book overwhelms their argument, both for those seeking to condemn Pius XII and Catholicism in general as well as those attempting to lionize the pontiff.

One of the few exceptions is Robert Ventresca's recent offering: Soldier of Christ: The Life of Pope Pius XII. His biography is remarkably mild in bias. Still, the subject of the Ustase only gets a few pages (261-267). He mentions that the Vatican did not recognize the Ustase regime but also did not publicly condemn it either. (262) The reasoning for that takes up about two chapters of the book, and it lines up largely with an earlier post of mine.

For a full account of Vatican-Ustase relations we will continue to have to wait. While the Acts and Documents of the Holy See Relative to the Second World War is an extensive 11 volume compilation of documents from the papacy of Pius XII, it is not exhaustive. Generally the Vatican library waits 75 years from the beginning of a pontiff's papacy to release his documents. With Pius XII being elected in 1939, that 75 year wait would give us the year 2014. Francis has said that the release will be "soon," but the Vatican's definition of that word differs greatly from the standard dictionary definition. Some historians (notably Phayer) allege that there will be damning evidence found in the unreleased documents. Others (notably Rychlak) dispute this. Until historians are granted access in the near(ish) future we won't know for sure.

I'll try to give a context both to ustaša genocide and to the documentary, as best I can. Important note - pointing out the spin in documentary itself, and later propaganda use of some tropes in it, is not in any way intended to absolve Ustaše, or deny a very real genocide and violence.

I can't find when the documentary was made, but I'm guessing its late 80s or early 90s. I haven't watched it, but it works with a figure of 700.000 killed, which is exaggerated and has its own contexts.

There is no way, and no serious researcher, accepts the figure of 700.000 Serbs killed. With some variations (ranging up to perhaps 10%, definitely no more), figures by Žerjavić are generally accepted concerning Yugoslav war dead. In total, about 530.000 Serbs were killed in Second World War, out of that in NDH close to 300.000. Clearly that was a genocide, and personally I don't consider numbers to be the most important matter in judging is as genocide.

The 700.000 number came about from (not so accidental) report on Yugoslav loses after the war, when total demographic loss (2.02 million) was reported as a number of war deaths, for purposes of reparations. As actual number of war deaths was 1.027 million, a difference had to be made up later. That is the origin of the inflated numbers. In late 80s and early 90s, they were used as nationalist propaganda at first, war propaganda later.

Another context is that of very real, and very violent genocide. It in large part came directly from ustaše ideology, a mix of fascism, extreme nationalism and the ultimate goal of making an independent and ethnically homogeneous Croatia, on an area much larger than that inhabited by mostly Croats. NDH included, among others, almost 2 million Serbs (in today’s Croatia and Bosnia and Hercegovina).

Politically, national relations were very tense in old Kingdom of Yugoslavia, with widespread feelings of resentment in Croatia over Serbian political domination and economic exploitation. To what extent those feelings were justified or grounded in reality is less important, crucial thing is that they were almost universal among Croats. Furthermore, assassination of Croatian representatives in Yugoslav parlament in 1928, (one of the killed, Stjepan Radić, was a symbol of Croatian nation), and of king Alexander in 1934, promoted a feeling among both Serbs and Croats that common state is impossible. When Yugoslavia was disintegrated and occupied in 1941, national conflict soon followed.

Back to ustaše ideology. One idea, never written down but quite well confirmed and followed in practice, was a doctrine of "three thirds": one third of Serbs in NDH was to be converted to Catholicism, one third expelled, and one third killed. Under NDHs own racial laws, Jews and Roma were singled out as undesirable (a "favour" to Nazis, that brought ustaše in power in the first place, but also consistent with ustaše own ideology), and than Serbs.

Violence agains Serbs can in general be divided in two categories. First is a "spontaneous" form, performed by so called "wild" ustaše [divlje ustaše], unorganised bands attacking Serbs without direct orders from the state, or sometimes even against immediate state policy. However, the ideology pointed above should make it clear that even if state didn't order all of violence directly it did set the stage and let it play out. There's no absolving ustaše regime from responsibility on that form. Another is more explicit and organised, and mostly done in concentration camps. Jasenovac is the largest and most notorious, where somewhere between 90.000 and 100.000 people were killed, Serbs in large part but not exclusively. Many Serbs responded by armed insurrection, first to save themselves, but some later affiliated themselves with Chetnik movement, that itself had its own genocidal programme against Croats and Muslims, but that's another story.

As for Vatican, I can only give a most general overview. It is not that Vatican was in favour of genocide, and it was only lukewarm at best regarding the legitimacy of NDH – at first. Catholic hierarchy in general was divided on the question of forced conversions. On one hand, it is clearly illegitimate, on the other it grew the number of Catholics. But, as during the war a communist guerilla, partisans, became the most prominent force in Yugoslavia, by 1943 definitely, the Vatican stance was that anything is better than communism. Support for ustaše regime therefore grew, with more or less dishonest expectation that fascist regime can be transformed into something more liberal.

The ethnic tensions between Serbs and Croats during the Kingdom of Yugoslavia (pre-WW2) were preceded by the bigotry and hatred of certain right-wing political parties in Croatia, as early as late XIX century (see Ante Starcevic and his proto-fascist ideology). The Serbian army, while liberating Croatia at the end of WW1 was not welcomed as a liberating army, but rather an occupying one. King Alexander failed to see it, or chose to ignore it!

Therefore, the brutality of the Ustase did not originate in these tensions between WW1 and WW2, but it was rather a continuation of the hatred that was perpetuated for many decades before WW1 and WW2.

Why? I really cannot tell. Serbian domination in political and economic affairs might have been one of the causes, but not the main one. I guess fear plays an important role in it. As does the hysteria which was omnipresent in fascist countries at that time. My humble opinion is that the Croats felt repelled by the similarities they have with the Serbs (genetics, language, traditions, etc.), which they saw as an obstacle to their path to become a westernized, "civilized" nation. To put it bluntly, Serbs were seen as backward, oriental people, a possible threat due to its strong cultural and religious links with Russia, and a menace to the Catholic church.

I will not comment the role of Vatican in the Balkans from WW2 to this day. Alojzije Stjepinac was sanctified a few years ago. Enough said. Also, I would consult the records of Israeli organization for the most correct number of holocaust victims in ex-YU (such as Yad Vashem). They have no reason to lie.