By a very strict legal interpretation, it would possibly be incest, because they would be related by affinity. They wouldn’t be related by consanguinity because they didn’t share any blood, but they would have affinity through marriage.

What happened with the marriage between B’s sister and A? If they ended up marrying other people and having children, they would have to have been granted an annulment, meaning their initial marriage legally never existed. There was no such thing as “divorce” in the modern sense. If they were divorced, that meant they were living apart but were still legally married, and they couldn’t marry other people. So let’s assume they had an annulment! How would that affect their future relationship and the relationship of their children?

I suppose they could have claimed that their marriage was never consummated, and that would solve all the problems here - in that case, no, this situation would not be incest. The actual sexual act created affinity, regardless of whether a couple was legally married (according to some legal theorists, at least). Perhaps, however, they received an annulment for another reason - they couldn’t have children, or one spouse was impotent, or if one of them hadn’t freely consented, for example. In that case an annulment would erase their marriage, but would it erase their sexual history? And if sex created affinity, would their children with other people still be related by affinity?

The canon law of the church had pretty complicated rules about who was related by consanguinity and affinity. From about the 7th century (or maybe even earlier) up to the Fourth Lateran Council in 1215, marriage was prohibited between people who were related within “seven degrees” (i.e., they shared a great-great-great-great-great grandparent). That made it very difficult to find someone to marry, especially for European nobles who were all related much more closely than that. In 1215 they changed it to four degrees, which made things a bit easier - now partners couldn’t share a great-great-grandparent. Of course for anyone with enough power/influence/money, it was easy to get special permission to ignore the prohibition, and they could also obtain an annulment if things didn’t work out. But that’s consanguinity. What about affinity?

“Pope Celestine III held that sexual relations between two persons created a legal affinity that barred subsequent marriage between either party and any close relative of the other-the affinity was similar to that created by acting as godparent to a child.” (Brundage, pg. 356)

Celestine died in 1198, and his successor Innocent III adopted this rule in 1215. So technically if B’s sister and A had any sort of sexual relations at all (married or not), they would have affinity with each other’s children. Noble A could not marry the child of B’s sister, and vice versa. Of course theoretically they wouldn’t be allowed to marry anyone else either, so let’s assume they got an annulment...although again that raises the question of whether an annulment erases past sexual history.

Your actual question is whether A’s children could marry B’s children, not the children of B’s sister, and the answer is…mmmaybe? It doesn’t seem like anyone thought of this specific situation in the Middle Ages. But based on the rules of affinity, they would still definitely be related, even if the exact relationship was a never defined in this way.

“According to the principle of the unitas carnis, the blood relations of a man' s wife were equivalent to his own blood relations and it was concluded that anyone who married one of this man ' s relations by affinity were also related to him. Thus affinity begot affinity . This relationship was known as the second class of affinity . If one man was related to another by this second class of affinity and the second man was left a widower and remarried , his wife would be related to the first man by what was known as the third class of affinity.” (Wahl, pg. 74-75)

The only somewhat similar example that springs to mind is Henry the Young King and Margaret of France. It’s not quite the same because they were actually more closely related than your example. Young Henry was the son of Henry II of England and Eleanor of Aquitaine. Margaret was the daughter of Louis VII of France and his second wife Constance - but Louis’ first wife was Henry’s mother, Eleanor. Louis and Eleanor’s marriage had been annulled because they weren’t able to produce a son, but Eleanor produced lots of sons with Henry II. So, Louis and Eleanor’s daughters were Young Henry’s half-sisters, and he married his half-sisters’ half-sister, but what does that mean for Young Henry and Margaret’s affinity?

Apparently that issue never came up. There were much more pressing issues - they were related by consanguinity much more closely than the prohibited seven degrees (this was 1160 so the rule hadn’t been changed yet). Even more importantly they were only 5 and 2 years old! They were younger than the legal ages of 14 and 12 and obviously they couldn’t agree to the marriage themselves. The marriage was also clearly politically motivated and intended to make things difficult for Louis. No one was too concerned with whether they were also related by affinity or not.

So the answer is probably yes, it would be incest, but probably yes, they could still get married. The affinity would be so subtle that only professional canon lawyers would have noticed, and even if anyone had objected, it was fairly easy to get special dispensation from the pope (as long as you hadn’t annoyed the pope in some other way first). For all practical purposes, A and B’s children probably could have married without any problems, but I would be interested to see if there any other similar examples to the contrary.

Sources:

Francis X. Wahl, The Matrimonial Impediments of Consanguinity and Affinity: An Historical Synopsis and Commentary (Catholic University of America, 1934)

James A. Brundage, Law, Sex, and Christian Society in Medieval Europe (University of Chicago Press, 1990)

Lindsay Diggelmann, “Marriage as tactical response: Henry II and the royal wedding of 1160”, in The English Historical Review, vol. 119, no. 483 (Sept., 2004), pp. 954-964