The question comes from a discussion about Hamilton on this sub. It made me wonder what the definition of this term is and if it is actually useful when thinking about the past before the invention of the ideology.
So, at its core, "proto-fascism" is the vague idea of 'fascism before fascism'.
Fascism did formally 'come into being' at the turn of the 1910s into the 1920s. The Italian fascists under Benito Mussolini took power in their famous March on Rome in 1922, and Mussolini formed his party, the PNF ("Partito Nazionale Fascista", National Fascist Party) in 1921. The term 'fascism' is in turn derived from the Italian 'fascio', which means something like 'federation' (as in 'a group of federates') or 'league'. In the Italy of the 1890s through 1910s, numerous such 'fasci' ('leagues') existed, and they were generally activist groups – often violent street gangs – with explicitly political goals.
Mussolini assembled his own 'fasci Italiani di combattimento' ('Italian fighting leagues') in 1919, uniting his ragtag alliance of political followers which he had assembled since his break from the Italian Socialist Party over the issue of the entry into World War I in 1914/1915. By 1923, the term "fascism" was internationally recognized as a phrase for Mussolini's movement, and leftist intellectuals had begun to apply it to other right-leaning authoritarian governments and movements that bore resemblance to the Italian primus.
So much for the history of how the term fascism came to be.
But there were people before "fascism" who certainly influenced fascism, even if they were not fascists themselves. (These are also not necessarily the "proto-fascists" you ask about!)
French aristocrat Arthur de Gobineau is one of the forebears of European scientific racism, identifying a separation of humanity into three key races into white, yellow, and black, with a clear descending line of primacy of human value between them. Gobineau's thought structure (which was actually quite pessimistic and convinced of inevitable racial decay) was later expanded by Vacher de Lapouge and Houston Stewart Chamberlain, who placed explicit importance upon the role of Jews as an internal 'anti-race' prone to subvert the white race identified by Gobineau.
German philosopher Friedrich Nietzsche was critical of representative democracy as well as the collectivist promises of socialism, instead repudiating such "herd psychology" and summoning in its place the "will to power", raising emotion and instict above its repression.
French engineer Georges Sorel elevated in his reflections on then-current Marxist movements the importance of violence in the achievement of political goals. Sorel's ideas not only applauded Lenin, but also found their way into Italian fascism through Italy's revolutionary militant syndicalists that joined the fascist movement.
As you can tell from this (not at all exhaustive) list of intellectual celebrities of the late 19th and early 20th century, much of the philosophical groundwork of the fascist ideological family (racism and antisemitism, totalitarianism, violence, masculinism, cult of leadership, militarism, etc) were certainly not invented by fascism.
As a result, other political thinkers were able to assemble some or all of these components before Mussolini and later Hitler rose to prominence.
It is these political thinkers as well as their followers that are usually pointed to as the proto-fascists, even though their classification as such is almost always controversial, as this is a classic example of a retroactive ascription by historians – i.e., the historian recognizes a pattern in his history and attempts to stretch it backwards, potentially towards a breaking point where it becomes meaningless in the timeframe into which it is transcribed.
Probably the most famous example is Charles Maurras, who in 1899 founded the neomonarchist French political movement Action Francaise. Maurras' movement was authoritarian, anti-Semitic, deeply Catholic, and sported a street movement similar to Mussolini's blackshirts or Hitler's brownshirts. The famous German Cold-War-era conservative historian Ernst Nolte sees in Action Francaise the "beginning of fascism".
Several Italian figures have been identified, such as the nationalist poet Gabriele D'Annunzio (famous for his military occupation of Fiume/Rijeka after World War I) and the Futurist Movement, a group of militaristic and nationalistic intellectuals around Filippo Marinetti. Marinetti later worked on the manifesto of the PNF, and both men remained prominent in Fascist Italy.
Other movements frequently described as proto-fascist include the Freikorps (nationalist-minded and anti-socialist paramilitaries in post-WW1 Germany, often consisting of army veterans), the Black Hundreds (monarchist, nationalistic and anti-semitic militias in Imperial Russia during the 1900s and 1910s), and nationalistic political parties/movements such as the Italian ANI or the German DNVP.
It is also worth noting that some authors use "proto-fascist" as a chronological term (i.e. a fascist-esque movement before 1919), whereas others use as a conceptual/evolutionary term (i.e. a movement that is on the way of becoming fascistic). In that latter sense, political movements of today could still be "proto-fascist".
Regardless, the question you ask in your post body remains: Is the term "proto-fascism" useful?.
If I can offer my personal opinion, the term is quite useful in the strict context of Fascism Studies. The fascist ideology did not arise from nowhere, and to have a term to identify chronological forebearers that were not-quite-but-almost-fascist is very useful. Likewise, its other definition, i.e. a political movement on the way to becoming fascist, can also be usefully described as proto-fascist.
But I think the term should not used in a vacuum. The statement "Maurras was a proto-fascist" is not very productive on its own (as it conveys little meaning and might even be dangerously misinterpreted by implying that Maurras saw himself as a proto-fascist), but the statement "Maurras can be classified as proto-fascist because his political movement combined elements later found in fascism, such as anti-semitism, political authoritarianism, and a willingness to mobilize street violence" can put the term into a proper political context.
In essence, the term "proto-fascist" should be contextualized, as the term 'proto' could mean 'distantly and coincidentally related' or 'on the very doorstep'. "How proto is proto?", if you will.
But that said, even if I did not find it useful, the term is rather frequently used by historians such as Roger Griffin and Ian Kershaw. So it is definitely worthwhile to have a good grasp of what the term implies.