Those "finger signs" are technically called mudra (jp. inzo 印相). They're strongly associated with Buddhism and are associated with a number of historical marti art schools (jp. koryu bujutsu 古流武術). That included folks who might be said to practice ninjutsu or schools thought to be ninjutsu schools. That history is complicated, so let's focus on the usage piece vs what is/is not "ninja."
From a historical perspective, Tenshin Shoden Katori Shinto Ryu, Itto Ryu, Shinkage Ryu, Shinto Ryu, Kukishin Ryu, and others all include kuji kiri 九字切り (lit. cutting the nine symbols) and typically involved the same nine characters you'd find in mikkyo 密教 (Japanese esoteric Buddhism). The nine characters are rin 臨, pyo 兵, to 闘, sha 者, kai 皆, jin 陣, retsu 列, zai 在, and zen 前. This is pretty common knowledge you can find in books and scrolls around Japan as it pertains to Buddhism; with the right hand (the hand representing heaven) you form a sword and cut a a grid while intoning (verbally or silently) the character in question as each one has a different meaning. The common overarching goal is to create clarity of mind, like a meditation of sort, almost like a vajrayana where the idea of enlightnment can happen in an instant. In some schools, details go down to how each finger represents an element and their position maps to ways in which energy flows work. It's a cool and complex topic.
Those historical schools would teach how to use the kuji in their own way, with different goals, focuses, visualizations, and so on. Unlike in anime and manga, as far as I'm aware in my studies, no warrior stopped in battle to invoke the kuji against their opponent in order to create a hex or exert what might be considered an unnatural influence on others. Pre-battle, post-battle, in solo study- would definitely have a place.
Other hand positions besides the typical ones in the kuji are used as well. Many of them are familiar to Buddhist practitioners, and while they too may have been used as part of a warrior's spiritual practice, there were practical applications that may arise from a mudra that would be used in combat. For example, one might not form their fingers into a position like this while waiting for an enemy, but having the fingers extending while holding swords to attack the face, grab, etc are part of the teachings, as would be to imagine yourself crashing down and engulfing your enemy like a water tornado (tatsumaki 竜巻) while you do so. The intent and mental attitude (a kokoro kamae 心構 of sort, like a "posture in the heart") would still be there. All of it done with the purpose of achieving an empty mind mushin 無心 or sometimes kushin 空心... though the latter really mean an empty mind with infinite potentiality in the moment. Of course, if you had to pretend you were a monk and needed to know how to attack, draw a weapon, and more from various hand positions, that would also contribute to learning how to use the hands in those capacities :)
Let me give you a passage from the late Otake Ritsuke of Katori Shinto Ryu on kuji and the like-
In bygone eras, warriors inscribed the kuji (nine magical signs) to their precious swords, tsuba (handguards), spears, and similar items, and the juji (ten magical signs) on the nakagno of their blades [part of the sword covered by the hilt]. In his book Nihon Toko Jiten (Swordsmith Dictionary), in the section on Koto swords, Fujishiro Yoshio mentions that nakago inscribed with the juji can be seen frequently in the creations of Hiroyoshi of the Yamashiro sword-making tradition. This type of inscription can also be found on the walls of castles, and was the physical embodiment of the devout prayers of the master of the castle that a vendetta be successful, that he may not suffer defeat in battle, or for a long life of martial fortune. They may also have been inscribed to ensure that the castle did not fall to the enemy. Alternatively, however, they may have been the part of psychological warfare on the part of the enemy's ninja, who could have infiltrated the castle disguised as laborers to conjure rebellion and chaos. In any case, the juji are an entreaty to personal deities to grant wishes or to achieve missions armed with a well fortified will.
Kashina Shin-Ryu may or may not have kuji, I don't know enough about that school, but Dr. Friday in his book Legacies of the Sword references inyo ittai 陰陽一体 (Yin-Yang as One) amongst other concepts which relates back to the physical manifestation of these esoteric ideas. Pointing this out as another example of how esoteric- including "finger signs"- manifested in warrior traditions.
So in short, the hand positions were reflections of spiritual Japan that warriors made use of in a myriad of ways as their teachings prescribed. I cannot confirm anyone casting spells with them in battle, though Otomo Komaro (the famous envoy to Tang China who also helped bring the famous Buddhist monk Ganjin to Japan) was killed in part for being an assassin and practicing witchcraft (maho 魔法). So who knows!
Sources-
