Whenever I watch a show or a movie set in the 1920s-1940s it seems like swing music and dancing were generally common in the United States. In American culture today there isn't a genre that everyone can dance to (except the ones that have instructions in the lyrics). So what happened?
Multiple factors, as is usually the case for any notable change in popular music styles. And keep in mind that swing music has never completely gone away.
Nat King Cole's Mr. Cole Won't Rock and Roll in a fun place to start when considering the decline of swing music in the US. As he put it in 1960, "One, two, three o'clock, four o'clock rock; you gotta sing rock or else you go in hock..."
So before we get to Bill Haley and His Comets, let's back up a bit to when swing music was still prominent. This type of music was played by big bands. Big bands had a lot of members: horn sections, woodwind sections, rhythm sections, conductors, and sometimes multiple vocalists. WW2 took its toll on these larger groups. Many musicians were called away to war. Glenn Miller, who was for a time the best selling musician in the US, famously disappeared in a plane over the English Channel while serving as a major in the US Army Air Force.
The large size of these groups also made touring fairly expensive and complex, compared to small, 3-5 member groups.
Recently developed electric instruments were also beginning to gain prominence and to be used as unique instruments in their own rights, rather than just as minor, amplified variations on older instruments. Link Wray's early use of just slight distortion was groundbreaking when Rumble was released in 1958; the sound was so unlike anything prior that it was banned from the airwaves. Increasingly playable solid body electric guitars and basses started edging out the older hollow-body electric/upright bass combo that had been the norm for decades.
Similarly, around the end of the swing era, the singer-as-star idea was beginning to take hold in a way that it hadn't really, yet. Prominent vocalists were featured, but the composer/conductor was styled as the real star. "Tommy Dorsey and His Orchestra Featuring Frank Sinatra" was one of the last forms of billing for that incarnation of the group*. (This was before, as the unverified legend claims, Sinatra's 'guardian angel', a mobster figure named Willie Moretti, stuck a gun in Dorsey's mouth in order to convince him to release Sinatra from the terms of his contract so he could go on to be a solo star.) In early rock band names like Bill Haley and His Comets, or Buddy Holly and His Crickets, you can still see the influence of the older styling of ensemble names.
Finally there is a generational aspect to it. By 1960, when Nat was singing about how he wouldn't rock 'n roll (but he could if he wanted to), swing music had become the music of parents and grandparents. As often happens, an upcoming generation will develop and embrace a music style uniquely different from that of the previous generation, perhaps as a means to develop a sense of identity.
Musician perspective coming in.
Dancing actually took quite a few more decades to decline than your question represents. The best explanation comes from two places: the economics of making people dance to music, and the technology of music, music composition, & instruments. For the most part it comes down to: how LOUD can we make this music, and how cheap is it to make it that loud while still sounding okay? However, the full history of the rise and fall of Big Band an dancing to live bands is a pretty big story, involving two decades of music technology advances, an act of congress, union salaries, and competition from other entertainment sources.
For now we'll ignore taste because it's difficult to say exactly what factors are driving the consumer market at any given time, especially since record companies and managers had quite a bit of influence over what music received exposure (via bribery and later vertical integration) at any given time that allowed them to set trends.
Additionally, the amount of jargon and specialized knowledge required for music means that oftentimes historians and cultural critics will over-emphasize non-musical aspects of music history that are more difficult to disprove ("the culture was ready for a change!") over much more practical explanations ("with amplification, a rock band can provide exactly the amount of loudness that a big band can, with about 12-18 fewer people needing to be employed, 12-18 fewer instruments requiring transport, and fewer specialized and expensive arrangers/orchestrators and copyists needing to be contracted"), or explanations that require further specialized data from psychoacoustics experts, musicologists, and several other groups of specialists ("The music - for a variety of quantifiable reasons - wasn't as 'groovy', so people slowly stopped going"). There was a massive generational shift, but I'll leave that for others to deal with; for now it's safe to say that, were swing bands more economically viable as loud dance bands than rock bands, the next generation probably could have found a way to get their own sound using the big band.
It's also important to understand why and how big bands came into being BEFORE we can explain why they declined, because a lot of this just comes down to technology and cost.
First we have to understand music technology. Prior to electronic amplification, a symphony orchestra is the loudest music-making body that we have that's capable of playing what we consider "songs". A large late romantic orchestra can play a little above 100 db, which is just below the loudness that a big band or a rock band can reach. However, a large late romantic orchestra is also often over a hundred people playing at once. Expensive!
Dancing Comes First
Now, dancing is fun, but it's also LOUD with all those people yammering and stomping their feet. The style of dance and the social norms of dancing (whether you can shout along with it, or whether your best bet is to speak under your breath) is pretty dependent on what instruments are available and how loud they are. Broadly speaking, in situations where the community sings and plays instruments, the dancing can be much more raucous, and when you have especially loud instruments - lots of percussion, banjos, accordions, and brass - you're allowed a more athletic dancing style. Quieter instruments (small string ensembles, guitars, mid-range woodwinds before the saxophone) require a more delicate dancing style if you want to hear what you're dancing to.
Remember that for most of human history, music tends to be used by cultures either for worship/contemplation, or for dancing, partying, that whole business. It's social, though, and active. Our culture of mostly passive listening is a relatively new invention, and a culture of passive, background listening that can be done alone is even more recent.
So when you're making music, you want it to be LOUD enough to be heard. Loudness is the key goal here, with musical demands coming right behind.
{For some source material, oftentimes drummers, arrangers, and other musicians from eras before DJs supplanted the role of dance bands often talk about the interaction between the music and the dancers, and how a live band responds to the changing dance styles (and vice versa). Perhaps the best demonstration of this comes from soul drummer Yogi Horton talking in this video.}
A few decades into the beginning of the 20th century, the dance band becomes popular, and with it the rise of jazz. This is made possible by a few miraculous inventions:
Technological Advances That Made Loudness Easier (And Which Gave Us Big Band)
In jazz harmony, the entire ensemble (minus the rhythm instruments - bass, piano, drums, sometimes guitar or banjo) is allowed to move parallel or similarly in the same direction as the melody in giant chords. This lets 4-13 instruments create a hugeness and ensemble prominence to the melody that previously was available only to much, much larger ensembles (getting strings to play as loud as a jazz brass section requires at minimum 48 people if you're minus microphones).
This century's worth of technological development leads us to The Big Band, detailed in the comments below.
Note: edited to remove some unnecessary tangents into music theory
Just because a fad was fairly widespread doesn't mean it was universal; many American cultures of the 1930s/1940s had their own musical traditions and their own types of dancing (or prohibitions against dancing). But moviemakers use certain types of music and activities as easy ways to set the scene, to quickly signal the era, the kind of dress, and the loosening moral strictures of the time. Similar shorthand: every 1920s show will include a speakeasy and maybe someone doing the Charleston; ones about the 1970s will show leisure suits and disco.
According to the prologue of Bob Stanley's excellent history of postwar pop music Yeah, Yeah, Yeah The History of Pop Music, two strikes led to the decline of big band swing music. First the American Society of Composers, Authors and Publishers (ASCAP) blocked any of that organization's songs from playing on the radio in order to push for higher royalties. These included songs by Irving Berlin, Johnny Mercer and Cole Porter. These songwriters could read and write the kind of big band music that was popular at the time. ASCAP's rival Broadcast Music Incorporated (BMI) was able to capitalize on this strike. BMI's music was mostly jazz, blues, and country which ASCAP had largely boycotted. So BMI was able to popularize those styles over big band music (interestingly songs with no American copyrights, foreign and old songs, also were produced and broadcast on radio during this time for the same reason).
In 1942 the musicians' union also had a strike which banned all live performances by their members. Singers weren't considered musicians, so a capella bands were put together to back stars such as Frank Sinatra. This allowed many singing stars to break out: "In the swing era--the big band years from 1935 to 1945-- the singers had usually taken second billing to band leaders; when the musicians' union finally negotiated settlements with the record companies, they found that the popularity of the bands had been largely eclipsed by their vocalists." (Stanley, xix)
This development obviously helped rock 'n' roll music begin as rock 'n' rollers were singers first and foremost, but to say that rock 'n' roll music displaced swing would be to ignore at least ten years of what Bob Stanley calls "pre-rock" pop music which is most certainly not big band or swing music.
This is partly an answer to your second question, as the singing stars generally made music that was not as danceable. This is a huge oversimplification and a bit of a cherry pick, but contrast bandleader Count Basie's One O'Clock Jump with Nat King Cole's Unforgettable. It might be more accurate to say that the function of Count Basie's song is to get you dancing while Nat King Cole's song is for listening or maybe dancing very slowly.
For a great explanation of the decline of dancing music in favor of listening music read the provocatively titled How the Beatles Destroyed Rock 'n' Roll: An Alternative History of American Popular Music by Elijah Wald. I would try to paraphrase an answer from his book but I'm still reading it.
Dance music has almost always been popular in the United States, it just keep morphing as one generation succeeds the other. Swing was a particular style of music with a particular style of dancing, but it was shoved aside by rock & roll, and its particular styles of dancing. Interestingly, Chubby Checker has made the claim that he invented "dancing apart to the beat" with his single "The Twist." He even went so far during the 90s to place full page ads in national music magazines and trades like Billboard, demanding the music industry recognize and reward him for this.
Since the swing era, about the only time that dancing to music seemed to fade was the early 70s, when rock music became focused on the artistic nature of the music - lyrically, compositionally, intstrumentally - and the listener was encouraged more to listen rather than dance. Much of that arose from the use of recreational drugs to enhance the experience, which could make remaining vertical a challenge. It is hard to imagine dancing to Pink Floyd's Dark Side Of The Moon.
Despite this, there was no denying that people still liked music's ability to bring the sexes together, and the lack of danceable music in the early 70s contributed to the blossoming of the disco era in the second half of the 70s. That was replaced by the unique dance styles of punk and new wave music in the late 70s/ early 80s, and eventually by rap and hip hop in the late 80s and early 90s. Now in the 21st century there is still danceable pop music and hip hop music, as well as EDM. If you don't think there is still dance music out there, then you haven't been to a club or wedding lately.