In the Middle ages, who/what decided if a country was a county, duchy, or kingdom? Were these titles anything more than just titles?

Mostly it was a mixture of tradition, the authority of someone more powerful, and giving yourself the most important title you could get away with.

In the Holy Roman Empire, for example, there were five original (or "stem") duchies: Saxony, Lorraine, Franconia, Bavaria, and Swabia. Over the years, however, more appeared. This was sometimes as a result of imperial grants, e.g. the Duchy of Austria. Other times they simply arose from minor nobles who decided to call themselves dukes at a point when no one else cared enough (and had the power) to stop them. At some points the emperor even created kingdoms, like Bohemia and debatably Poland. The general force of this was a sort of title-inflation over the centuries. Being a duke in the HRE meant you were really important in 1000 or so; in 1500 maybe not so much.

For a contrasting example, consider England, a country which, for most of its history, had much better central leadership. After the Anglo-Saxon conquest, it was divided into seven minor kingdoms; they occasionally called one of the kings the Bretwalda (a sort of hegemon), but did not have central rule. This changed when most of them were destroyed by Vikings; Wessex was the only kingdom to successfully resist the invaders, so it took over the rest of England. From then on there was only ever one king in England, barring civil wars. Even dukedoms were often reserved for the Crown's close family members--as opposed to the dukes of the HRE, who caused no end of trouble and tried to keep the emperor weak. When English dukes clashed with their king, it was generally not because they felt their oats and wanted greater independence, but because they disliked the king.