There are a number of details that complicate your question. First off, around where? Alchemy was practiced in a variety of ways in a variety of places, taking advantage of developments in material and changes in philosophy that happened over centuries, and so alchemy in the eighth century is different than alchemy in the fourteenth century.
The next complication is the way in which alchemy was conceived. The underlying assumptions about the natural world that the alchemists, and their contemporaries, held were very different than the ones we hold today. The Aristotilean notion that the world was comprised of four elements (earth, air, fire, and water) had accompanied with it an entire system that explained how the natural world operated. Metals, for example, were viewed as water-based (due to their liquid-like nature when heated) with an essential earthiness. At the same time, there was the idea that gold was the most perfect form of metals, and that given enough time all the half-formed metals would ripen into the perfect metal. You can see this understanding of the nature of formation with regards to people in different literary works: the hunchbacked and deformed Richard in Richard the Third. He is "Cheated of feature by dissembling nature, / Deformed, unfinish'd, sent before my time / Into this breathing world, scarce half made up" (1.1.19-21). Generally speaking, the alchemist's job was to find a way to accelerate this development. The believed the notion was sound and possible because it fit with what they thought of the world. That there was also textual and traditional authority relating to the art (writers like Albertus Magnus, one of St. Thomas Aquinas' teachers, wrote on alchemy and its possibilities) lent credence to the art as a whole.
While the end goal was always the philosopher's stone, the pursuit led to the development of different technical apparatuses and chemical discoveries, and a related refocus on activities other than the stone. Things like the "Green Lion" (aqua regia) which was quickly adopted into alchemical procedures because of its ability to dissolve Gold – again, the king of all metals. Other developments, like fractional distillation, allowed for the creation of pure alcohol (aqua vitae) in the thirteenth century – which was taken as the purest element, the ether, the stuff that the stars and planets are made of – which was used as a medicine because of the idea that it could draw out the curative essences of plants and flowers. Amusingly, this is the origin of Chartreuse, that fine alchemical libation.
Then you have the difficulty of the textual tradition, and its relationship to different occult philosophies. For example, the writings of Hermes Trismegistus – the name given to the author of a number of different texts dating to the first few centuries AD – were viewed as particularly important, if opaque, instructions for how to make the stone. However, a great deal of effort was spent struggling over the meaning of things like the emerald tablet. If the alchemist failed to create the stone, then you it might have been a simple case of misinterpretation of the text, and so further attempts were necessary. That others, like Theophrastus Bombastus von Hohenheim, often tried to redefine the operations behind the stone's creation (specifically, he talked about the need for an essential salt, in addition to the more traditional refinement of sulphur and mercury) served to complicate matters.
Lastly, there was a spiritual aspect tied to the occult nature. If the alchemist failed, then he might have failed due to his spiritual impurity, and so he must undergo rigorous penance so that God might find him worthy. There are common stories of an adept teaching a novice a little of the art, and giving him a little piece of the stone, which the novice then misuses. After a chastisement, the novice then spends the rest of his life working back towards a completion of the art. The parallels with the story of the Eden (innocence, Fall, and then a long period of work towards restoration) are obvious, and they speak towards the way in which alchemy was tied to a Christian spirituality.
So, to summarize, alchemy was never subject to a mentality that said "try it once, and if it doesn't work, move on." It was far more complicated than that. If you're interested in further reading, the Alchemy Reader is a good resource, even if they're a bit credulous at times regarding who wrote what. And I'm always happy to answer more questions.
I'll try to break this down in some sort of logical fashion.
it was believed that it worked. As in, legends abounded of people actually having discovered the philosopher's stone or elixir of life. Including people like Albertus Magnus, Thomas Aquinas, and Nicholas Flamel; none of who were even alchemists. It was therefore just a matter of finding out how they did it.
It was always debated whether it was possible at all. There were several philosophical as well as empirical reasons for believing it was possible. I talk at length about this on my podcast. For instance whether transmutation happened in nature, if so, was it possible to replicate, or even improve the process. And when it happened in a lab (because they thought it was done before), was it purely natural, or was something supernatural involved?
As mentioned in other comments, the scope of alchemy depends on who you asked and when. Some alchemists were very open about just gilding metals, or shining metals to look like silver, or creating an alloy to look like gold. And in no way was this charlatans. They were very open as to creating fake precious metals and why.
Alchemy didn't fall by the wayside until it was proven several times in several different ways that it couldn't possibly work, and even then it took a century to die. People simply had two theories: alchemy is real; and alchemy is not. Without tons of evidence both sides had reasons for believing. I can go into that a in more detail if you want, I've mentioned it many times in other answers. I'm on vacation with limited internet, so please have patience with my replies.
Some still believe, just snoop around the kookier corners of the internet to find alchemy alive and well despite modern atomic theory. Or just visit /r/alchemy (no disrespect to the fine folks over there).
Basically alchemy lasted some 1400 years. But even the cutoff date is arbitrary. Atomic Theory was published in 1803. By that time the scientific community had long given it up, but even then there were alchemists. They still existed in the 20th century. The reasons given as to how alchemy is possibly always changed, but the belief in alchemy never really died until the 18th century (and not even then).
Shameless plug: I wrote a 20-or-so page book that I'm selling for a buck that answers all your basic alchemy questions, like what they believed, why, who, how, etc. The podcast is free ;) ..or just keep asking here. As for sources, my books are literally on the other side of the planet right now, but I may be able to recall some of it off the top of my head.