I'm going to begin this answer by saying (like in many cases of dealing with ancient Rome) that the situation was far from clear cut. As always, I'm going to have to include a little context of the evolution of client states and the differences between them. Please remember that this is a much discussed (and therefore much disagreed on) part of Roman political dealings and therefore the more reading you do around the subject the more of an idea you'll get of the differences. Particularly between east and west. It may also be helpful to have a map of the Roman Empire in front of you! Each client state was unique. Unfortunately for us, that means there is no one complete answer that will satisfy all aspects of your question. I'll give a few famous examples that may help though. So, to begin with, in terms of Empire Rome had two well-known political divisions: provinces and, as you ask, client states. These were all born originally out of the Romano-Greco tendencies to colonise different parts around the Mediterranean. Hence some of the foundation of Rome stories (See Virgil's Aeneid). Rome originally began as a city-state that had to compete against others including the likes of Albalonga (in Latium). I'm going to break your question down into examples. 'Eastern' kingdoms and some British examples. The reason being that, to the Romans the east and west did almost form very distinct blocks. This is reflected in different governing styles and the differences in the culture and history. Even under the conquests and annexations of Trajan (when the Empire reached its greatest extent, see handy map) The east remained politically fragile. British Examples (main source = Tacitus):

During the consulship of Lucius Caesennius Paetus and Publius Petronius Turpilianus [AD 60-61], a dreadful calamity befell the army in Britain. Aulus Didius, as has been mentioned, aimed at no extension of territory, content with maintaining the conquests already made. Veranius, who succeeded him, did little more: he made a few incursions into the country of the Silures, and was hindered by death from prosecuting the war with vigour. He had been respected, during his life, for the severity of his manners; in his end, the mark fell off, and his last will discovered the low ambition of a servile flatterer, who, in those moments, could offer incense to Nero, and add, with vain ostentation, that if he lived two years, it was his design to make the whole island obedient to the authority of the prince.

Paulinus Suetonius succeeded to the command; an officer of distinguished merit. With this view, he resolved to subdue the isle of Mona; a place in habited by a warlike people, and a common refuge for all the discontented Britons. In order to facilitate his approach to a difficult and deceitful shore, he ordered a number of flat-bottomed boats to be constructed. In these he wafted over the infantry, while the cavalry, partly by fording over the shallows, and partly by swimming their horses, advanced to gain a footing on the island. - The Annals, Tacitus (AD 110-120), Book XIV

As is evident from the text above. The direction that a client kingdom could be taken could depend massively on the personality and ambitions of local governors. Not necessarily the wishes of the emperor. One such example where this went terribly wrong was in the case of the Iceni under Prasatagus.

Prasutagus, the late king of the Icenians, in the course of a long reign had amassed considerable wealth. By his will he left the whole to his two daughters and the emperor in equal shares, conceiving, by that stroke of policy, that he should provide at once for the tranquility of his kingdom and his family. – The Annals, Tacitus 31

From this we can surmise that there was an economic and political agreement as the Iceni’s position as a client kingdom. We know from accounts of the invasion that the Iceni were not protagonists in resisting Roman advance. Therefore, Prasatagus must have formed a working relationship with the new Roman system. He keeps his throne and his lands on the condition that he either pays a ‘tribute’ to Rome (since they cannot be taxed, as they are not formally part of the empire). This benefits both parties and the Romans have peace and don’t have to bother with immediately installing new administrators for the area, which keeps the locals in check too. In theory it’s a great system…

The event was otherwise. His dominions were ravaged by the centurions; the slaves pillaged his house, and his effects were seized as lawful plunder. His wife, Boudicca, was disgraced with cruel stripes; - The Annals, Tacitus 31

We all know how that one eventually ended. But, it does show how fragile the role of ‘client kingdoms’ were and how individual rulers both on the Roman and native sides defined the individual political situation at any one time. Another example from Britain is the case of Queen Cartimandua, who ruled over the Brigantes (a loose conglomeration of tribes that covered most of present day Yorkshire, Lancashire, North east and west) 51 BCE she handed the ‘resistance leader’ Caratacus over to the Romans. Tacitus writes quite a bit about the goings on in Brigantian politics, mainly because the Romans became involved in order to sure up Cartimandua’s grip on the throne. The hilly and sparsely populated region that Cartimandua controlled would not be conquered and annexed until Agricola served in Britain. So having a friendly ruler meant the Romans could spend time ensuring the peace and prosperity of southern Britain before taking on more conquests. Cartimandua was eventually rescued by the Romans in 69 CE when she had attempted to replace her husband with her armour bearer. What became of her isn’t known, but it does indicate that client kingdoms where more of a two-way street than it might appear from the outset.

The east is characterised by its complexity. On this point, as you say it is fair to understand that "the states were essentially subservient to Rome without actually being part of the empire" It would also be fair to add the caveat that it is far better and easier to let a people be ruled by their own established monarch than trying to introduce a whole new political system. One example (now mired in current world news) is Palmyra. It was an important trading post and Rome certainly became very involved in its history. Before Palmyra became part of the Empire (under Trajan) it formed a sort of intermediary. Neither part of Rome or part of Parthia, but known to and interacted with both. Part of the Palmyrean ability to maintain such a tenuous position between the two superpowers was the fact that is was in the middle of the desert. Thus making it very difficult to conquer. Ask Mark Anthony, he tried in 41 BC. What is interesting about Palmyra is, after it became a province, it staged a very nearly successful break from Rome under the infamous Queen Zenobia. Who, as the widow of the Roman serving Odenathus was on paper initially an unlikely leader of a break-away cause.

Another more famous example is the kingdom of Judea. The most famous king being Herod. In this case the Roman’s allowed the Jewish peoples religious freedom. The famous census (that meant certain significant peoples had to make the commute from Nazareth to Bethlehem). The Roman understanding was that as long as Herod was able to keep control of the people, paid due tribute and allowed the Romans to exercise certain powers, then he, and his family, wold remain in control. For Rome, local kingship was an easy way to keep large areas under control. Josephus mentions how Herod sent his sons to Rome in part to meet the emperor (Augustus) but also Mariamme stayed in the imperial court (23 BCE) (Josephus, Antiquities, 15.342-3).

You also mentioned the idea of a ‘buffer’ state to keep the peace. The most important example of this was Armenia. The Kingdom formed a small division between Rome and Parthia. By collaborating on no should be crowned king (Parthia was meant to suggest candidates, Rome ratify them) it was intended that the two superpowers could keep a distance between each other and avoid unnecessarily antagonising each other over an already tense border. Again, this worked well in theory. I’ve already rambled on enough, but the fact that the period from 92BCE can be included in the phrase Roman-Persian wars, tells you how it turned out as a long term policy.

The states themselves would be very unlikely to identify themselves as ‘client’ and each one differed to the next. If you were interested in some more another very interesting case is Commagene. There were certain traditions, like sending sons to be educated in the imperial court. This simultaneously stood as a guarantee of obedience and meant that the next generation were brought up understanding and appreciating Roman style of governance. But as we saw with Palmyra, this could dramatically backfire too. For the states themselves it might be said that it was more along the lines of political and economic expediency to bow to the wishes of Rome. But that didn't mean that they remained so, or thought that challenging the rule of Rome was impossible.