Experience, built up from trial and era. This continued well into the industrial era, the maths of stress and strain in materials were only fully established by Augustine-Louis Cauchy, a French mathematician, in 1822. And even then, engineers were slow to adopt the techniques, particularly outside France. James Edward Gordon, author of the fantastic book, Structures: Or Why Things Don't Fall Down, and one of the founders of material science, dates the start of the use of mathematics by French engineers to the early 19th century, with British and American engineers following from about the 1850s. Gordon defends their caution: engineers would apply a hefty "safety factor" to the basic calculations, and even then find that structures would fail from time to time. Why was only really explained by the development of the mathematical understanding of cracks, in the late 19th/early 20th centuries, with work by a number of mathematicians and engineers - Alan Arnold Griffith's theory of critical crack lengths in brittle materials was only published in 1920, and the theoretical expansion of this to all materials was done in the 1950s.

Luckily for builders of castles and cathedrals, the main stresses on their structures were compressive - pushing in - rather than tensile - stretching or pulling or twisting. Rocks and bricks tend to be very strong in withstanding compressive forces. Gordon calculates that a wall of bricks could be as high as Mt Everest before the bottom rocks would be crushed under the weight.

Therefore the main failure mode in a stone or brick building is that one or more of the walls will topple over, probably from the forces of the roof. So if your castle or church or other building design worked at a small scale you could scale it up to a bigger one and it would probably work. Probably. Medieval cathedral builders were stretching the limits of this approach, with their desire for height and sticking windows everywhere, and would run into problems if, say, the ground they were building on settled more in some places than others. But this wasn't as big a problem for castles because the builders wanted to build the walls thick enough to defend against rams, catapults and eventually cannons. So the basic design was stronger.

Pre-industrial-era builders did have materials available that were strong against tensile forces - wood and ropes (as trees have to stand up to winds), and iron and steel. But while rocks and bricks are common, these tensile materials had supply issues. Iron and steel were very expensive to produce in quantity before the technological improvements that were part of the industrial revolution, and large pieces of wood were (and are) rare, as they take centuries to grow.

Bridge builders could use arches, a concept which dates back millienia and which convert tensile forces into compressive ones. This meant bridges could be built out of widely available stone or bricks. There are however problems with scaling up arches to cover larger and larger distances, particularly if you want ships to be able to pass underneath. And arches aren't any help for ships or aircraft.

Therefore bridges, ships and early aircraft were built conservatively, and even so there were a number of failures and near failures.

I very much recommend Gordon's book. It's from 1978, but it was recommended to me by two separate civil engineers and it's brilliantly written, it explains ccomplex concepts simply and funnily.