This is an older answer, taken from this thread.

Explanations for the mechanism behind the burning of stars only really became necessary once the concept of conservation of energy was developed. Before then, it was felt that the sun might burn forever, in line with the then-dominant idea of 'uniformitarianism'. With the sun around forever, explaining why it burned was felt to be more the realm of philosophy than physics.

This changed as physicists began to explore the field of energy conservation, in the late 1830s and early 1840s. Most astronomers did not assume the Sun was burning some kind of chemical. Experiments in the late 1830s showed this to be impossible. Measurements of the luminosity of the sun suggested that it would have to be burning at a rate difficult to achieve with chemical fuels; equally, the lifetime of the sun should it be burning a chemical was inconsistent with estimations of the age of the Earth. As such, these ideas were quickly discarded. Another idea, discarded on similar grounds, was that the sun had been heated at an earlier time by some other event, and was slowly cooling.

The first published theory for the source of the Sun's energy was published in 1848 by the German physicists Julius Mayer. He had considered and discarded chemical processes. He also attempted to explain it using the Sun's rotation as a source of energy, but again found this to be insufficient. Ultimately, he decided that the sun must be powered by infalling meteors. The gravitational energy of these meteors provided the energy for the sun to burn. This required a considerable amount of mass to be present; Mayer calculated that between 94 × 10^12 and 188 × 10^12 kg would need to be falling onto the sun every minute. In 1853, this idea was independently developed by a Scottish physicist, John Waterston, who gave a talk on the topic at a meeting of the British Association (Mayer's paper wasn't published in English until 1863). Waterston did not develop his theory in detail, but it was adopted by William Thomson (later Lord Kelvin). Thomson put Waterston's ideas onto a more quantitative basis, publishing a paper on the topic in 1854. He calculated that the Sun would accrete about 100 Earth masses in 4750 years, a rate which Thomson did not consider impossible. In this paper, he would also attempt to calculate the age of the Sun; he estimated that the sun had been burning for 32,000 years, and would likely continue burning for another 300,000 years. Thomson attempted to use the discovery of the anomaly in the perihelion of Mercury as evidence for the meteoric theory; however, it soon became clear that there would not be sufficient infalling mass to create the anomaly. Ultimately, the theory was dead by 1870, as observations showed that there were not enough meteors present to power the sun.

The next major theory had actually been suggested by Immanuel Kant in 1785, though it had been almost totally overlooked. Kant, noting that gases heated up when compressed, suggested that gravitational contraction might power the Sun. Waterston alluded to a similar concept in his 1853 talk. The next year, Hermann von Helmholtz (who had attended Waterston's talk), gave the idea a more sound theoretical footing, announcing this at a lecture in Konigsberg. This idea played well with Helmholtz's preferred idea for the formation of the sun, which was that it had formed from the collapse of a larger nebula. By the 1860s, Thomson had slung his weight behind the gravitational collapse theory, believing that, while the meteoric hypothesis had been responsible for early heating of the Sun, the current state of the Sun was down to gravitational collapse. The Helmholtz-Kelvin theory became generally accepted. However, there were two key failings. Firstly, there was a lack of observational confirmation of the key mechanism; every contemporary observation of the sun suggested that its diameter was constant. Secondly, and more importantly, it clashed with the expectations of biologists - evolutionary theory required the Earth to be a few hundred million years old, while the gravitational collapse theory suggested that it the Sun was a few tens of million years old. There were a few competing theories; the Scottish geologist James Croll suggested that the Sun had formed, and been heated, by the collision of two large bodies moving at high speeds, while the German industrialist William Siemens suggested what was, in essence, a solar greenhouse effect (the Sun was, in this theory, surrounded by gases that were both heated by solar radiation, and drawn back into the Sun, where they decomposed, giving out light and heat, before being ejected by the Sun's rotation to reform).

The discovery of radioactivity in the late 1890s and early 1900s provided another explanation. Based on the assumptions that the composition of the Earth and Sun were similar, and that all matter was, to a greater or lesser degree, radioactive, many physicists assumed that radioactive decay powered the Sun. This concept was further supported by the vast quantities of hydrogen in the sun; this was interpreted as coming from alpha particles formed by radioactive decays. The radioactive decay theory drew a large amount of support from notable physicists, including Rutherford, Soddy and Poincare. However, its days were numbered, as spectrographic observations of the Sun and other stars showed no evidence for elements known to be radioactive. By 1915, it was essentially discarded. Most physicists now believed that stars were in some way powered by some form of nuclear energy; however, no effective mechanism for releasing this energy was known. The discovery of nuclear fusion provided the key.