Well, the simple answer is of course that in general they weren't 'enforced', at least not in the sense of being subject to some sort of supranational legal procedure. It's a bit more complicated than that, though.

To a large extent this is more of an IR question than a history question, since you're basically asking the abstract question of how equitable inter-state relations can emerge given the power asymmetries inherent to any real-life international system. It strikes at the heart of one of the key issues that has historically divided the competing schools of IR theorists. Because of this you might be better off asking someplace like /r/asksocialscience.

(As an aside I don't think the existence of the ICJ and the UN necessarily changes that much in any case -- neither of them have state-like powers of enforcement.)

I can provide a more strictly historical answer, though, from what's often called an 'English School' perspective.

Essentially, from this perspective, the answer lies in the fact that no two states exist in a vacuum. Relations between states are inherently articulated through a cultural prism that goes beyond a single specific exchange.

A variety of localised international systems have prevailed throughout history. These have ranged from the ritualised rotating hegemony of the Sumerian city-states to the Confucian tributary system that revolved around imperial China and the often confused and contradictory idea of Christendom and the Empire in medieval Europe.

In each of these various cases, we find that instead of being an indeterminate state of nature where anything goes in relations between states, international relations are structured by important underlying conventions and cultural ideas. These ideas are what make something like the very concept of a 'treaty' feasible and comprehensible. This becomes obvious in interactions between different international systems, which are often difficult and confused: the unsuccessful British missions to China in the 18th and early 19th century are a case in point.

The upshot of this is that states have historically tended to define and behave according to certain conventions that regulate their relations with each other. All states share some stake in maintaining their overarching system of relations because they each directly benefit from the mutual trust and political certainty that derives from it, not to mention the personal moral obligations that state leaders will feel.

In Europe, for example, there's the idea of a jus gentium, a 'law of nations' which articulates the conventions that all peoples can be expected to abide by regardless of their individual state laws. This idea is a tradition that goes back to ancient Rome. (Nowadays we also distinguish a jus inter gentes or 'law between nations' that applies to things like treaties, but that's largely an 18th-century invention.)

From ancient Roman thought through to medieval canon law, anyone stepping outside the bounds of jus gentium -- such as by violating a treaty -- was by definition violating natural law at a very profound level, and could thus expect to provoke moral outrage and distrust.

The Romans themselves, as Cicero's De officiis suggests, saw the upholding of promises as one of the most fundamental aspects of a moral society. Cicero gives one instructive example:

Again, if under stress of circumstances individuals have made any promise to the enemy, they are bound to keep their word even then. For instance, in the First Punic War, when Regulus was taken prisoner by the Carthaginians, he was sent to Rome on parole to negotiate an exchange of prisoners; he came and, in the first place, it was he that made the motion in the Senate that the prisoners should not be restored; and in the second place, when his relatives and friends would have kept him back, he chose to return to a death by torture rather than prove false to his promise, though given to an enemy.

As far as Cicero was concerned, this was clearly the right thing for Regulus to have done: Rome's reputation was at stake! (Bear in mind, too, that De officiis was for a long time the second most widely read book in Europe right after the Bible, and remained widely read by statesmen at least up until the 19th century.)

So at the end of the day, yes, where an international system breaks down in some way or conventions are ignored, what matters is who's stronger. But treaties have historically emerged in the context of overall systems of moral and political norms which determine how the governors of various states view each other, and how they can expect each other to act.

Like I said though, this is only a rather superficial answer to something that's a very far-reaching theoretical question. For more on this particular analysis, and a more general history of international systems, you might be interested in Hedley Bull's book The Anarchical Society.

edit: I realised I answered the second part of your question ('why did people make treaties if they couldn't be legally enforced') but not the first: what, short of war, did people do about rulers who overstepped the limits of convention? This is unfortunately much more complicated to answer -- the answer obviously depends on what place and time you're looking at. As a quick example though: In medieval Europe, for instance, failure to comply with a treaty was typically treated as swearing an oath under false pretenses -- i.e. perjury -- and thus a matter of ecclesiastical law. So, other than simple war, the full suite of ecclesiastical sanctions could apply to a ruler who broke a treaty -- sanctions which would be exacted by the papacy. (I'm not aware of any cases where a whole kingdom or town was put under interdict for violating a treaty, though; a medievalist might be able to clear that up.) This is the sort of 'moral approbation' you might expect from my explanation above.