Great question. Hope you don't mind a departure, but I want to set the scene a bit with some history of why Shuttle came to be in the first place, what it was replacing, and why it was expected to be an improvement before answering the question itself.

So here's the issue -- spaceflight is SUPER expensive. For example, Project Mercury spent more than $2 Billion in current money to send 6 crew into space, only 4 of which going into orbit. By the time Apollo was done, that had ballooned to the modern equivalent of nearly $200 Billion.

Now granted, a lot of that is initial investment like R&D as well as maintaining support staff during down periods, tooling, innovation expenses like building the hardware to move the rocket components, etc. But each mission still had a significant cost that was draining the coffers of the treasury at the same time as the Vietnam War was in full swing. The moon mission popularity varied wildly, from about 34% in 1968 prior to the first flight, peaking after the successes of Apollo 11 and 12, and by the last mission being back in the mid-30%. Over 60% of Americans polled in 1971 believed NASA's moon missions were a waste. So Congress cancelled Apollo missions entirely, slashed NASA's budget, and forced them to change tack.

There was still hardware ready though, so NASA scrambled starting in 1966 (when long-term funding first started getting tight) to develop was was known as the Apollo Applications Program, which could be additional use-cases for the Saturn rockets already developed. The idea being that if they were told they couldn't got to the moon anymore, or the funding made insufficient to do so regularly, that Saturn rockets may still have enough uses to deem their continued development and manufacture worthwhile. NASA was really reluctant to lose their only rockets and potential raison d'etre, as they might be eliminated entirely before another rocket system could be developed.

Out of the Apollo Applications Program came 2 missions that actually did fly -- the Apollo-Soyuz Test Project (ASTP) and the Skylab Program. Keep in mind, the US and the Soviets have been in the Cold War for 2 decades at this point, and the competition is fierce, so actually meaning what they were saying about wanting to use spaceflight as a means for peaceful exploration and science, the US and the Soviets began discussions about actually doing a short joint venture in Low Earth Orbit, a location that both sides had rockets that could deliver crew safely to. The Soviets liked this idea because it helped them regain a bit of clout after having lost the race to the moon, and the US wanted the additional positive PR and flights. The other project, Skylab, was a repurposed fuel tank which was outfitted as a living area for crew for long periods in Low Earth Orbit. The Soviets had had a number of space stations prior to this point, but all were fairly temporary and small. This would be 22 feet in diameter and 85 feet long -- an absolute monster! With all the extra lifting capability that the Saturn V had (lifting to Low Earth Orbit is a lot easier than to the moon), they could outfit it with large telescopes to study the sun constantly, research hardware that wouldn't fit in a capsule, and even use the interior space for testing their spacewalking maneuvering system that was in development. Congress approved both missions just months after Apollo 11, but limited the number of rockets that could be built after that. All said and done, there would be one Saturn V (heavy lift rocket) available for putting Skylab into orbit, 3 Saturn 1B (crew launch vehicle to LEO) available for taking crew to and from Skylab, and one Saturn 1B for ASTP. That's it.

Now that's not to say NASA didn't have other plans in the works. They could read the tea leaves and knew Congress wanted something cheaper and more sustainable. So they dusted off some designs that von Braun had developed almost 2 decades earlier and began work on a Space Plane. It utilized a large drop-tank like long-range fighters at the time did, and could reach orbit, return, and be refueled and reflown in just a month. Congress heard the pitch and funded it, announcing the development during the final Apollo mission. While this meant that Shuttle would never be used for a moon mission itself, it could be loaded with cargo modules to build any mission architecture you wanted in space for a fraction of the cost and schedule. With four shuttles and a turnaround time of four weeks, NASA was excited about all the new missions that would soon be available and with the lower budget it was still within reach.

So now we get to the heart of your question: why did it become seen as a boondoggle and what kept it from reaching its potential. Well, as every engineer will tell you, every decision you make in a design comes with a cost.

Safety was a serious issue with Shuttle, so let's start there. Originally, the design was deemed too expensive for NASA's new limited budget, so they had to go ask the Air Force for partnership. The USAF agreed, but levied their own requirements on it to ensure it was able to perform any missions they may need it for. One of the big ones is the addition of massive wings. These were heavy, reduced payload to orbit, and created more friction than the original design, but the USAF wanted to be able to do a launch and recovery from the west coast of the US in a single orbit, which meant it had to be able to sway 15 degrees (how far the Earth would rotate under it while in a single orbit) back as it came through the atmosphere again -- this is called cross-range capability. NASA didn't have any such need or desire, but without the USAF funding they were stuck, so shuttle got bigger and heavier. Next came safety designs -- in Mercury and Apollo, the vehicle came with a Launch Abort Tower (the very tip of the rocket) which had a motor in it that could pull the capsule away from the rocket in an emergency and high enough for the parachutes to fully deploy. Gemini did away with the LAT, opting to use ejection seats for aborts low enough that the parachutes wouldn't have time to deploy. Shuttle originally carried both designs, ejection seats for the crew in low altitude abort scenarios and a separable crew cabin with parachutes for higher abort scenarios that wouldn't allow a return-to-base or abort-to-orbit. This traded weight and complexity for safety, as studies had shown the likelihood for needing those abort options to be very low. Both of these decisions ended up costing crews, the Columbia due to larger and more obtuse wing angles from the cross-range capability, and the Challenger due to no separable crew cabin recovery. This is a significant reason why Shuttle ended up getting axed.

But there were others, of course. Let's talk schedule. Shuttle was chosen as the successor to Apollo in 1970, with contracts going out shortly thereafter, but due to the sheer size and complexity of this program it took 6 years to build a test model that could be used for testing, and it wasn't until 1981 that we had the first launch. Skylab had been left in Low Earth Orbit at the end of the Skylab 3 mission in 1973 with the plan to return with the first Shuttle and continue using it, but due to the massive shift in schedule, Skylab's orbit deteriorated enough that it completely deorbited in 1979. So instead of starting with a massive, already-outfitted space station and just needing to design new vehicles to get there, NASA now had the vehicles and nowhere to go.... and no way to replace the massive space station that had just come crashing down!

Plans were quickly drawn up for a replacement. Revision after revision were made, first under the name Space Station Freedom, then Space Station Alpha. In the meantime, we had to either bring our own small space station along in the back of the shuttle (SpaceLab), or work with the Soviets to use theirs (Mir). Finally, it looked like a design was nearly ready and the unthinkable happened -- the Berlin Wall fell and the USSR collapsed. Suddenly there was a real concern in Washington DC: there were thousands of Soviet missile and rocket engineers who had been working on peaceful space exploration weeks ago, who were now jobless and might be swayed to go help a rogue nation like North Korea or Iran build up a missile program of their own. The US shifted their space station plans once again, opening a channel for international partnership based primarily with the Russians and Europeans, and in exchange both NASA and Roscosmos could split the expenses and weather their financial straits. This became one of the biggest shining moments for the Shuttle -- finally it had a large mission to perform over the next 10-15 years...