This is really comparing apples vs oranges. The difficulty with comparing sword fighting styles is that each one was created to overcome completely different obstacles at very different periods of time.

Let's take the examples you gave; Japanese sword fighting was designed to to cut down a massively armoured opponent on a chaotic battlefield. Rapiers on the other hand were typically used in highly organized duels with those involved wearing little more than their every day clothing.

However, Rapiers were created in response to firearms becoming more prevalent. Guns were really shitty back then so you wanted to have a back up weapon, but guns were still guns so armor fell out of fashion. Since no one was using armor, all you needed was a small, light backup sword once you spent all your shots, effectively replacing the large, heavy swords of the medieval era.

This was also the same period when the Sengoku period was ending, and Samurai warfare also fell out of fashion. Again due to firearms.

So in a sense they were equally effective for what they were designed to do. If there were more angry, screaming samurai in 16th century italy I'm sure that more italians would have picked up Wakizashis, and if guns had been invented Japan, there would have been more thin swords and a lot less samurai.

EDIT: Another thing to keep in mind was that when rapier use became common, swords had not been primary battlefield weapons for several centuries. While swords were used as backup weapons in battle during this time the Pike, Halberd, and Polearm had long replaced the sword as the go to weapon for warfare. The gun only helped this.

Interestingly European style fencing and Japanese style fencing (kendo) did briefly meet up in the late 19th early 20th century. Many samurai found themselves out of work as Japan modernized. With their talent in martial arts some of them found their way into travelling shows. These traveling shows made it around the world to The United States and Europe. Kendo included some wrestling and other aspect that made it difficult to have just a "sword fight" but there are some records of European fighters and weapons against Japanese fighters and weapons. It was friendly demonstration between different styles so there really wasn't a winner or loser. However, the indecisive nature of the competition indicates that one isn't really more effective than the other.

I found some brief articles that cover some of the interactions

Portugese sailors did, in fact, duel a handful of samurai.

It went poorly for the japanese. The rapier is THE pinnacle of unarmored duelling swords, and very hard to compete against.

Against the closest equivalent-the longsword-the katana itself is likely to be a little wieghtier, have an inferior guard, and will actually be used in a manner that is more similar than different.

Two handed cutting swords are two handed cutting swords. You won't see much wild variation in usage.

Which era of European sword fighting are you referring? Medieval? Viking Era? Renaissance?

How do you define "effective"?

If you want a counterpoint against rapiers I'd look up George Silver's Paradoxes of Defence. He was extremely critical of the rapier as a self defense weapon citing duels where a man with a broadsword suffered multiple rapier wounds but was still able to kill his opponent as well as an instance where a foreign fencing master drew his sword on some dockworkers only to be beaten by oars and stretchers. However he was something of an english nationalist so his assessment may not be completely fair.

Really I think it comes down to whoever has the most experience. All other things being equal I'd favor the weapon with the most reach.