Why was medieval illustration (especially in calligraphy) so shitty by standards of other periods? Was it only because it was made by (essentially) clerks who did it as an afterthought of calligraphy, or was there more to it?

Your initial assumption is correct. There were several reasons for why medieval illustration in manuscripts was ... simplistic.

As you said, the illustrators were clerks or monks who were firstly responsible for copying the books. Each such a clerk was essentially a copist, illustrator, introligator and had to take care of the manuscript/volume he written. Without a strong division of labor, there was no way one could get "good at illustrating" unless he already was tallented.
Prior to the 14/15 century, the antique ideas of artistic techniques (knowing the correct proportions of the human body, undestanding of how anatomy dictates positions of the body parts, rules of perspective etc) were forgotten lore. This meant htat even if one illustrator or another KNEW how to draw/paint, say, human body in motion, or perspective between background and foreground objects, the skill would probably die with him.
Limitations of the medium. Vellum (and early paper) is a poor surface to paint on. It takes time to absorb some paints and inks/atramentum, and absorbs others too fast, forming a stain. Whats more, the surface of a piece of vellum/medieval paper would be unewenly absorbant, meanign that one square inch of it would drink ink, while another would let it run. Also, vellum was extremely pricy, and usually there would be not much space on it to illuminate, since the text would take at least 1/2 of the surface. A combination of this factors meant that painting, say, a detailed face with thin, delikate lines would probably not work, or worse, lead to a run or a stain. Therefore a crude, "comic-book style" of illustration was prefered.
Limitations of time. Illuminating a books was a workshop/manufacture style of work. There was no time to waste on excessive details(since an illuminated work would already have enough of them), since writing and illuminating a copy was already a very, very long and careful process.
Limitations of tools and utensils. Clerks quillbox would contain a few goose quills, one or two iron quills, a sharpened reed, a small knife and a stick of lead to draw lines. None of which would easily allow one to draw anything but strong, wide lines and curves (which was a necessity anyway, since thin lines and delicate writing would fade to easily after some time).
The reason why books were illuminated, was not to show artistic skill or beauty, but to illustrate a story. The symbolic message of a illumination was far more important than its artistic merit. For example: in St.Gregory's Manuscript, there is Gregory himself, Bebo of Seone and Emperor Henry II, as well as God's hand reaching from the sky, with a blessing. In the picture, Greogry and Bebo are more or less of equal size, but the Emperor (much more important person in Gregory's view) is almost twice as large. Similarily, the hand of god hints at even greater posture of the Almighty, the God's fingers themelves are bigger than Gregory's entire arm. The point of this is not that St. Gregory could not draw in proportion, but rather than it was more important to revere God and Emperor by making them larger in the picture.