Was there ever an instance of a tower being too short/tall come the assault and being unusable?
Generally speaking, armies that would take siege of a castle would understand enough about the technology of surveying to be able to determine the height of a castle wall using a device such as a mariner's astrolabe. Even in the case that they didn't have proper trigonometry charts, there was understanding of an approximation of the trig functions needed.
Astrolabe technology continued to improve through the middle ages, but the math needed had been around since the ancient world. About 550 AD the Christian philosopher John Philoponus wrote a treatise on the astrolabe in Greek, which is the earliest extant Greek treatise on the instrument.
Math students generally get exactly this problem to solve either in geometry, or at the very least, precalc, if not both.