The short answer is that Austria-Hungary was looking to establish itself as the hegemonic power in the Balkans. I'm sure you're familiar with the term "sphere of interest," and it would be accurate to describe Austria-Hungary's goal in the Balkans as such. In 1866, Austria-Hungary is defeated by both the Germans in the Austro-Prussian War and the Italians in what is called the Third War of Independence. Both of these defeats are serious blows to Austria-Hungary's Great Power status. Essentially, Austria is booted out of a position of influence vis-a-vis the soon-to-be unified Germany and Austria-Hungary's Italian possessions, including Venetia, is ceded to the Italians. Basically, Austria-Hungary is seen to be weak militarily and sees its powerful position in Central Europe taken by the Germans.

As a result, many Austro-Hungarian policy-makers decide that Austria's only area for expansion, and thus ability to regain its lost Great Power status, is in the Balkans. The Ottoman Empire is perceived as weak and dying (I've written about the pitfalls of "decline and fall" narratives before, so be careful here) and its possession of much of the Balkans is seen as tenuous at best. After the Russo-Turkish War of 1877-1878, Ottoman Turkey cedes almost all of its European possessions. Romania, Bulgaria and Serbia all establish themselves as independent kingdoms. The province of Bosnia & Herzegovina, though occupied by Austro-Hungarian troops, remains under the titular control of the Ottoman Empire.

The countries that establish themselves in the wake of the Ottoman defeat are very important to the story of Austria-Hungary. Remember, the Balkans are where the Austrians are going to regain their pride and lost Great Power status. They want to establish themselves as the regional hegemonic power. What would that look like? Well, all the land not controlled directly by Austria-Hungary would be held by countries friendly to the Austrians in terms of trade, military affairs and diplomacy. Much of the time between 1878 and, say, 1908 is spent trying to win these new countries over to Austria-Hungary.

Austria trying to establish itself as the big guy on the block on the Balkans necessarily brought it into contact with Tsarist Russia, which, consequently, was also trying to establish itself as the big guy on the block in the Balkans. Russia wanted to be the protector of the new countries, especially Serbia. Russia, similar to Austria-Hungary, had lost its Russo-Japanese War with the Japanese in 1904/5 and saw the Balkans as its new area for expansion and influence.

Austria-Hungary and Russia actually have a pretty comfortable relationship, even in the Balkans, until 1908, when Minister of Foreign Affairs, Count Alois Lexa von Aerenthal pulls a fast one over on the Russians and their Minister of Foreign Affairs, Alexander Izvolskii. Remember when I said earlier that Austria-Hungary had occupied but not annexed Bosnia & Herzegovina in 1878? Well now, Aerenthal decided he would go to the Russians and ask them if they'd be willing to support Austria's annexation of Bosnia and Herzegovina. Austria-Hungary and Russia, remember, were kind of cooperating, or at least tolerating, each other's interests in the Balkans. Well Izvolskii and Russia, despite the fact that their little brothers, the Serbs, had designs on Bosnia, allegedly give Aerenthal and Austria-Hungary the go ahead to annex Bosnia (make it a full and legal province of the Austro-Hungarian Empire) as long as Austria would support Russia in its attempt to lay claim to Constantinople and control of the Bosphorus Strait. I say allegedly because Aerenthal and Izvolskii's meeting was very secret and private and no minutes of the meeting survive. After the meeting, Austria annexes Bosnia but then fails to help Russia take the Bosphorus; this makes Russia look weak to the rest of the world, since it's well known that Russia is also looking at the Balkans as their special little area of interest. From this point on, 1908 to 1914, Austria-Hungary and Russia's relationship deteriorates.

So now Austria-Hungary is on Russia's bad side. Meanwhile, the king of Serbia, Alexander I Obrenovic, is assassinated in 1903. His dynasty had been pretty friendly with Austria-Hungary, but the new king, Peter I Karageorgovic, is decidedly anti-Austrian. The Serbs were a new nation in Europe, having becoming independent in 1889 (see above) but many Serbs were still living under Austrian rule in various Austrian provinces: in Croatia, in Hungary, and, since 1908, Bosnia & Herzegovina. The annexation of Bosnia & Herzegovina really pisses off the Serbs. They see Bosnia as Serbian land and, like the Russians, are incredibly upset when Austria annexes it. Three years later, in 1911, Serbia teams up with Greece, Romania and Bulgaria to attack the Ottoman Empire, since the Ottomans still have some European possessions. The Balkan League, as it was called, manages to defeat Turkey and push them out of the Balkans. Bulgaria gets a ton of new territory as a result. The other members of the Balkan League get jealous and in 1912, they team up and turn on Bulgaria. During this Second Balkan War, two Austro-Hungarian corps mobilize in a show of force against the Serbs. The Serbs ignore the threat of Austro-Hungarian intervention, aimed at intimidating the Serbs and keeping them from making territorial gains and coastal territory. Serbia wins. These two Balkan Wars end with Serbia looking strong and poised to be a regional power in the Balkans, challenging the hegemony of Austria-Hungary. Importantly, remember, Serbia is also tied to Russia. The rivalry with Serbia as a new upstart in the Balkans provides the context for the assassination of Archduke Franz Ferdinand in 1914, with which I'm sure you're familiar. The rivalry with Russia for influence in the Balkans provides the context for the assassination of the Archduke sparking a general European War instead of simply a Third Balkan War between Austria-Hungary and Serbia.

As a side note, Aerenthal and other Monarchists saw foreign policy as vital to Austria-Hungary's future. The Empire was a hodgepodge collection of over 11 ethnic groups, and the domestic political situation seemed stagnant to many contemporaries. There was a coterie of diplomats and soldiers in the upper echelons of Austria's decision-making circles who felt that foreign policy was a good avenue to sort of divert attention away from stagnant internal politics. Conrad von Hotzendorf, who would famously lead Austria-Hungary's troops in battle in 1914, many times suggested preventative attacks (preemptive strikes) against Italy and Serbia.

As one final note, Alison Frank at Yale is working on a new book about Austria-Hungary's imperial ambitions prior to the First World War, so if this topic interests you, look for that to come out soon. I gave you the short and dirty version of the story, so if you're unclear about any of the parts (some of them are rather confusing) or would like book recommendations on the topic, please don't hesitate to ask. I think it's awesome that you're looking into this period and Austria-Hungary specifically, and I'd like to help.

TL;DR - having been defeated in 1866 and not possessing any overseas colonies, Austria-Hungary needs a place to expand and exert its imperial influence. The Balkans are seen as fertile ground, but Russia and a new Serbia stand in Austria-Hungary's way.