So the best guess in the late-19th century was developed by Lord Kelvin and Hermann von Helmholtz, and is called the Kelvin-Helmholtz mechanism. Both were pioneers in the new science of thermodynamics, of which one of the consequences is that energy sources must be accounted for: you can't have energy coming from nowhere, which makes one then ask, so where is it coming from?

Combustion could be easily ruled out — there was just not a way to imagine there being enough fuel to sustain such a reaction for very long. They knew the rough size and mass of the Sun; it's just not enough fuel to imagine it being something like a hydrogen-oxygen reaction.

The Kelvin-Helmholtz mechanism instead said it was about gravity and cooling: the surface of the Sun was thought to cool, which would lower the internal pressure, which would cause gravitational contraction, which would cause heating. It's not a dumb idea, and in fact actually is something that plays a role in how gas giants work, and is a real effect in things like stars.

The main problem with this is that something as energetic as the Sun could not possibly use this to operate for billions of years. There is just not enough energy. Kelvin used this to argue that the Sun was only several million years old (to the frustration of Charles Darwin, a contemporary, who did not think that gradual evolution could operate on so short a timescale). But evidence emerged over this period that the Earth was billions of years old.

This contradiction is what led people to search for other plausible mechanisms, but for quite a while there was just simply nothing better out there. The English astronomer/physicist Sir Arthur Eddington referred to it as "an unburied corpse" in 1920, which I find pretty amusing. In the same lecture, Eddington gave an overview of the situation at the time:

What is the source of the heat which the sun and stars are continually squandering? The answer given is almost unanimous—that it is obtained from the gravitational energy converted as the star steadily contracts. But almost as unanimously this answer is ignored in its practical consequences. Lord Kelvin showed that this hypothesis, due to Helmholtz, necessarily dates the birth of the sun about 20,000,000 years ago; and he made strenuous efforts to induce geologists and biologists to accommodate their demands to this time-scale. I do not think they proved altogether tractable. But it is among his own colleagues, physicists and astronomers, that the most outrageous violations of this limit have prevailed. ... If the contraction theory were proposed to-day as a novel hypothesis I do not think it would stand the smallest chance of acceptance.

Eddington said that the new science of radioactivity and sub-atomic particles could point the way forward, and proposed in the same year that nuclear reactions, specifically fusion, were more likely the source of the energy. 1920 is much earlier than many people expect (because we tend to think of fusion and fission in technological terms, and nuclear fusion technology is much later than this); later work by others (notably Gamow and Bethe) paved the way towards a deeper and precise understanding of the processes.

The above account comes from the book of a dear friend: Nasser Zakariya, The Final Story: Science, Myth, and Beginnings (University of Chicago Press, 2017), which spends quite a lot of time talking about Kelvin and Helmholtz, notably the difficulties that were had in reconciling it with other forms of scientific estimates about the age of the Earth (it is a book about how scientists develop and think about grand narratives of time and development, so things like Darwinian evolution and the Big Bang play a large role in it). I bring this up merely because one might not realize at first that something as seemingly specific as "how does the Sun work" could lead into deep controversy and debates in fields as seemingly separate as biological evolution and geology.

(And yes, history of science is definitely within the focus of this subreddit! There are a few historians of science on here, like myself, who are regular contributors.)