How acceptable was it for a King or Emperor, for example, to strip a vassal from all his titles and execute them? Or were they just executed and allowed his titles to pass down to his issue? In which cases or how often were they shown mercy and allowed to go on with renewed oaths? What was the way to go for a liege who found his vassals conspiring against him or his family?
So first off, I just want to point out that the "feudal contract" as we envision it (a reciprocal contract between vassal and lord exchanging land for military service) may not have actually existed in the way we used to think it did during the Middle Ages if at all, and if it did, was definitely restricted in geography and time. It's part of the recurring series of debates on whether feudalism existed and whether it's a useful phrase at all. I'll avoid getting too deep into that subject, but we have links to previous talks about it if you're interested.
Now with that said though, depending upon the era and location, a vassal who rebelled against his liege would be treated a variety of ways.
For example, in 6th and early 7th century early medieval francia, because of the power of royal authority and the late Roman framework the Merovingians inherited, nobles who unsuccessfully rebelled were likely to be killed and have all of their land confiscated, and likely gifted out to other more loyal nobles as reward. This of course, occurred quite frequently, and was itself part of the lingering legacy of Roman administration, with the power of Merovingian kings as de facto Roman sovereigns.
But as you move to the 9th century carolingian era and especially through to the 10th century ottonian era, as nobles began owning land that was more consolidated in a local region (due to the civil wars of the late carolingian era), there was an understanding that even if a noble caused a lot of shenanigans, and had to be imprisoned or even executed, that the core home territory of that noble should still belong to the family, and thus be passed down upon his death. By this time, it would cause great political concern as well as instability for a king to seize land from his nobles in the way that his forebearers would've done so without qualms.
It was simply a matter of the breakdown of the wide ranging former legacy power of Roman administration, as it switched to the more localized politics that we all know and love of the high medieval era.
Which means by the time the high medieval era (circa 1000 CE) rolls around, it was not too uncommon for rebellious vassals to be shown mercy and just continue on, as given the lack of resources of the public state in this time, the same vassal who was rebelling against his liege, would be the same vassal the liege later relied upon to supply soldiers or police that same territory.