Yes, in medieval canon (Church) law, PIV sex outside marriage was a crime, whether adultery or fornication. A rough guideline is that adultery involved breaking a marital bond while fornication did not, but in practice the terms were quite flexible. A married man having an affair with a single woman might result in an adultery charge for him and a fornication charge for her, or two adultery charges; other times, a married man and a married woman having an affair would find only the woman charged with a crime--or vice versa!

Basically, as with most things in the Middle Ages, it's really important to look at local patterns in a specific time and place (assuming there is evidence). Otherwise, you might go with: although fornication was technically a less severe crime and sin than adultery, prescribed punishments were generally similar. However, the charges might not be levied equally against both involved parties.

Ruth Mazo Karras, Doing Unto Others: Sexuality in Medieval Europe, is generally a good introduction to matters like this. She also has some individual essays that look more closely at terminology and at court cases (for the latter, Unmarriages might interest you).