Most complaints about the annexation weren't actually about the annexation itself but about the way it was carried out, as you already mentioned - from a legal standpoint it was a clear violation of the Treaty of Berlin from 1878 as well as the Austro-Ottoman protocol of 1879 that explicitely stated that the occupation was only provisional, and it was done without prior consultation with the other Great Powers. It also has to be said that the uproar wasn't necessarily universal, for example the response of France to the annexation was relatively calm.

Now, the fact that the treaty of Berlin allowed mere occupation had several implications; most notably that the Empire couldn't fully include the population of Bosnia and Herzegovina in its political system (eg. it had no own constitution and didn't have its own Diet like other provinces of A-H) and there was also a theoretical chance that the Ottoman Empire would demand those territories back.

To elaborate, A-H did not really have the situation in Bosnia entirely under control, it had many difficulties in administrating the provinces, it had to maintain a sizable garrison to prevent uprisings and unrest like the one that happened right before Bosnia was occupied, the financial and skilled manpower expenses on improvement of infrastructure and bureaucracy was immense, which was financed by heavy taxation of the local populace, the peasantry in particular while the Austro-Hungarian administrative leaned on political support from wealthy Muslims and the bureaucracy was often staffed by catholic Croats; there were also problems with Serb, Croat and Yugoslav nationalisms, the A-H administration tried to promote more or less artificially created "Bosnian" identity, that did not take root however save for the upper class muslims. This financial burden caused unrest among the locals and even amplified the social and national problems and tension Bosnia had prior to the occupation. So the Austro-Hungarian control of Bosnia was somewhat shakier than it may seem.

One way to reduce this tension and consolidate power in both Bosnia and the Balkans in general would be through complete political representation in the Empire but that would require a separate constitution and a Diet; this idea popped up in the mind of Aehrenthal, the foreign minister of A-H already in 1907. Creating both could however reasonably constitute a violation of the nominal rule of the Sultan regardless, so to implement constitution, Bosnia would have to be annexed one way or another.

Regarding the possibility of a return of those provinces to the Ottoman empire - as I already mentioned the chances of that actually happening were more or less theoretical, though there was a possibility that the Young Turks, that carried out a successful revolution in 1908 which was also the pre-text for the annexation of Bosnia, could try to restore sovereignity of their rule over Bosnia after the consolidation of their goverment; afterall the Young Turks created a new constitution that would have to apply to Bosnia as well, so the implications were certainly there.

As to why it was a blow to Serbian ambitions: Well, as I mentioned the rule of A-H was still not fully consolidated and relatively shaky, so the chances of some pro-Serb uprising in Bosnia was not entirely out of question; also beside the nationalist concerns for Greater Serbia, Bosnia also represented a good shot at gaining sea access for Serbia.

However the biggest concern for Serbia were the implications of the annexation - From Serbian perspective A-H was consolidating its position in the Balkans and thus limiting Serbia's diplomatic and expansionist capabilities and Serbia was afraid of further A-H expansionsm in the region. This concern was not necessarily baseless, as A-H statesmen (Aehrenthal and Conrad von Hötzendorf in particular) considered the annexation itself a solution to the "South Slav question"; however considering the cost of administrating Bosnia, Aehrenthal basically dismissed any further territorial expansion in the Balkans as the Monarchy simply did not have the resources for it.

Sources:

RAUCHENSTEINER, Manfried: The First World War and the End of the Habsburg Monarchy, 2014

SKŘIVAN, Aleš: Císařská politika. Rakousko-Uhersko a Německo v evropské politice v letech 1906-1914, 1997

KODET, Roman: Austria-Hungary and the Ottoman Empire during the Bosnian Annexation Crisis 1908-1909, in Prague Papers on History of International Relations, 2009, p. 289-297