Have boomerangs ever actually been used as weapons?

Many, many times. The boomerang is mostly a hunting weapon. It saw less use against humans, but was also used in war. For both hunting and war, it was sometimes used as a sword-club (i.e., a club with a narrow sharp striking edge) rather than being thrown - this was more common in war.

However, the weapon-boomerang is the straight-flying boomerang. The returning boomerang is largely a toy (although apparently saw some use for hunting). Returning boomerangs are lighter and will be less effective on impact (though can still be effective for small game, such as birds), and present a rather more challenging problem for aiming. They do not appear to be effective weapons except against flocks of birds.

The physics seem quite impossible.

The physics can be divided into two parts: the aerodynamics and impact. The aerodynamics gives a straight and flat trajectory, out to 100m or so, providing more range, and more accuracy vs moving targets, compared to a non-aerodynamic hand-thrown projectile like a spear (and the range is not lacking compared with woomera-thrown spears (woomera = spearthrower, atlatl)).

As for effectiveness on a target, a straight-flying boomerang of typical weight (say, 400-500g) can deliver 100J or more to a target, which is sufficient for effect on emus and large kangaroos (but perhaps it isn't a good idea to try to hunt buffalo with a boomerang).

Isidore of Seville (Saint Isidore, lived c. 560-636) wrote about the boomerang as a Gallic weapon:

Clava est qualis fuit Herculis, dicta quod sit clavis ferreis invicem religata, et est cubito semis facta in longitudine. Haec et Cateia, quam Horatius Caiam dicit. Est genus Gallici teli ex materia quam maxime lenta; quae, jactu quidem, non longe, propter gravitatem, evolat, sed ubi pervenit vi nimia perfringit. Quod si ab artifice mittatur, rursum redit ad eum qui misit.

or, in English, via Google Translate,

The club is like Hercules, because it is tied to each other with iron, and it has a half inches in length. These and other commodities, which Horace Caiaia said. There is a kind of matter, which is especially a tough weapon of the Gauls; which, 's throw of all, a good way off, because of the weight most rapidly, but, where it reached to excessive fragmentation. But if he shall be cast by the artisan, and again he comes back to the one who sent me.

Isidore describes it as reinforced with iron, and this is supported by archaeology, with a 2,000 year old boomerang found in France appearing to have been iron-reinforced: https://exarc.net/issue-2015-3/ea/gaulish-throwing-stick-discovery-normandy-study-and-throwing-experimentations

For more discussion, see https://www.reddit.com/r/AskHistorians/comments/8us02r/how_effective_was_the_boomerang_as_a_weapon/