I have used counterfactual history in discussing with other researchers, but I would never put it into a paper, or attempt to seriously work out what could have happened.

As an example of how it could be useful:

in 1420 Henry V signed the Treaty of Troyes, which made him the heir to the Valois crown, and de facto ruler of France in the absence of a mentally stable Valois king. (This is simplifying somewhat).

Unforseeably, Henry V died in 1422, leaving his infant son Henry VI the crown of England and (on the subsequent death of the Valois king), the crown of France. What occurred (again simplified), was this. Henry V's brother, the Duke of Bedford, spent the next decades attempting to secure and extend English holdings in France. He was starting from a fantastic position, particularly since the disinherited dauphin (Charles) was unpopular, and since the immensely powerful Duke of Burgundy was an ally of Bedford's and of England's. He did reasonably well, but within thirty years England had lost France (Joan of Arc's intervention commonly held up as the reason), which was (I believe) one of the pivotal factors of the subsequent war of the Roses.

The question is, was the Treaty of Troyes sustainable? Could Henry V really have expected to win and rule the whole of France permanently? Was the fact of Henry VI's minority or flaws inherent in the treaty more important in determining the future of the French and English crowns?

We discussed whether we thought that, had Henry V outlived his French counterpart, he could have ruled both England and France.

This is an exact example of what you discussed above. We were trying to work out what structural and personal factors led to the loss of France, to Joan of Arc's successes, to the War of the Roses. This would never go into any of our research or anything- but as a thought experiment it was useful.

Having said this, I don't do it systematically and it certainly is mostly speculative.

You might be interested in the responses to a "Monday Methods" thread on this that was done a couple months ago. https://www.reddit.com/r/AskHistorians/comments/42nhdo/monday_methodsaccounting_for_choices_not_made/

I found all three responses to be quite insightful, particularly that basically all historical analysis rests on an implied counterfactual for any claims of importance or causation.

I think it can be a useful tool if used properly. But it has to remain a tool and not a goal onto itself - which is the perception that often surrounds counterfactual history. When most people hear counterfactual history, they probably have no idea what it is. But when most of those who do hear the term, they'll be more likely to think of the infamous 'What if Hitler had won the war?' questions rather than how counterfactual history is and should be used.

As you indicated in your question, the focus and end-goal of counterfactual history has to lie within "factual history". /u/perroybartlett gives a very good example of counterfactual history done properly. The end result of his research was always insight into the "structural and personal factors led to the loss of France, to Joan of Arc's successes, to the War of the Roses."

Moreover, it's also important to stick close enough to factual history to avoid your work becoming, as you said, mostly speculative. You can take "factual history" and change a fact and consider how this would have changed events, but it becomes more tricky to then continue speculating beyond the results of this original change. You also have to be aware of what those original changes are. It's very easy to make unsupported assumptions even within a relatively minor change - but this can really influence the legitimacy of your research.

To give a simple example: Jack kicked a football and it broke his mother's window. A counterfactual history could consider what would've happened if it was Jim, Jack's brother who was also there, who kicked the ball and broke a window. The fact that has been changed was the person who kicked the ball and broke the window. Within this change, there are already some assumptions - namely that we assume that Jim would've also kicked the ball towards the window, that he was able to kick the ball hard enough to break the window, etc. It's important to realise this because the way you constructed the change has already prevented you from gaining certain insights. Let's say, for instance, that Jim was a malicious little kid, always in trouble, on his mother's last nerve, while Jack's a little saint. We can assume that the punishment doled out by their mother would be different for each child. But because of the way we constructed our counterfactual, we can't simply say that it was for the better with regards to harmony within the family that Jack was the one that kicked the ball. After all, we're ignoring the possibilities in our assumptions. Maybe Jim wouldn't have hit the window at all.

Is our counterfactual useless then? Not necessarily, if we recognise its limits. If we're specifically interested in the family dynamics of our little family, we can still gain some valuable insights. We can, however, run into problems again if we start moving away from our original change and our original assumptions and start speculating further. There's a limit how far you can go, essentially. If you find yourself saying 'Well, if Jim kicked in the window and his mother punished him harshly, what effect would this have had on...', you're quick on your way to some real questionable terrain. If you move beyond a few reasonable steps beyond reality, it's difficult to retain legitimacy because you can't assume all other facts will conveniently remain as they were before you made your original change - it's like some form of the butterfly effect. Every step you take becomes more and more complex.

I say all this to show just how difficult it is to go down the counterfactual route. You really have to know your subject matter and you have to know exactly what you're doing. That's why a counterfactual like 'What if Hitler had won the war' or 'What if Germany invaded Britain' are somewhat absurd and can at best give some insights on some very limited parts of history.

That said, I do think it's a very good way to avoid a rigid view of history - falling into the trap of teleological history or historicism. We're all very tempted to think that history happened in a certain way because it was meant to happen that way - to the point where we forget or neglect the fact that everything could've been very different. By adding the possibility of something else having happened, you're giving yourself some perspective in this regard.

So I personally think it can be a useful tool, but it's to be handled with care.

The biggest problems with counterfactuals are the butterfly effect and what we might call the "inverse butterfly effect."

The butterfly effect: change one thing and LOTS of things start to change at once with it, in wholly unpredictable ways. The answer to "how would history have changed if Hitler won" is "No one can possibly know how 1946 would have been, so say the least of 2016."

The "inverse" butterfly effect: most things in history have multiple converging causes, in other words, positing a counterfactual about one things is effectively positing a counterfactual about many things, probably too many things for the counterfactuals to be useful. Hitler lost for a lot of reasons, and for him to have won you would need many of those reasons all to go away at once. More oil, smarter generals, a better plan for Russia, keeping the Japanese from pushing the Americans into the war, a more effective submarine force such that the British and Americans couldn't establish such domination of the seas...