Getting it "right" wasn't something that was particularly important to them. The important thing was engendering the proper reaction in the mind of the viewer. So you want to communicate that Christ is a king, you dress him like what every one knows that a king looks like. You want to communicate that Joseph of Arimathea was wealthy, you dress him like a wealthy person so your audience gets it immediately. A medieval thinker knew that the world didn't "really" look like this, but they weren't trying to create an accurate cartographic representation. Instead they were trying to reveal fundamental truths about the world, such as the fact that Jerusalem lay at its center.

Part of this is simply a result of different emphases, as I've indicated, realism in the modern sense didn't really exist. However, there are deeper, metaphysical reasons as well. Within the generally Neoplatonic understanding of medieval thought, there's a sense that the intellectual understanding of an object is more real than any physical representation. Thus, it was more important to generate the proper understanding, which in artistic terms was done through a highly elaborate symbolic system that they were inculcated with since birth. You could see an Ox and immediately think "oh, the Gospel of Luke" or a set of keys -St. Peter, a Pelican - Christ, and so on. I'd love to give a specific source for this, but really it's just the whole of medieval thought. You see it everywhere, from histories to hagiography and from icons to world maps. Mary Carruther's work on memory and rhetoric offers a fascinating account of how monks and scholars would train themselves in this symbolic system, as does Leclerq's Love of Learning and the Desire for God, and you can see the metaphysics that I've described in any mystical text I can think of, Bonaventure's Itinerarium would be a good place to find it.