The game of "football" has a very long and complicated history. In the beginning, football referred to a collection of games that was played on foot (hence the name). There wasn't a formal set of codified rules.
In the early 19th century, each school had their own set of rules. So the game played at Yale would be different then the game played at Princeton for instance. Similarly, the game varied across the different English schools, so schoolboys at one school would play a different game then those at a second. Attempts to standardize the game started around the 1850s in England, with rule sets such as Cambridge Rules and Sheffield Rules. Association Football (also known as soccer, short for association) was formalized in 1963 and the Football Association was created. In 1871 The Rugby Football Union was created to formalize the Rugby Football rules. So therefore all modern footballs can essentially trace their creation to the 1860s and 1870s. (Aside: Australian Rules Football was created earlier pretty much independently in Australia in 1859 when the first set of rules were formalized, hence why it is so different to the other codes of football).
in 1873, a few american colleges came together to attempt to standardize their rules of football. They were mostly based on the Football Association rules (so early soccer). Harvard did not participate in this standardization process, so they had difficulty playing against other American schools. Instead, Harvard decided to play against McGill University in 1874. McGill played a type of football similar to Rugby, and Harvard took a liking to it. Eventually Harvard (who played a rugby style game) challenged Yale (who played a soccer style game) in a set of compromise rules. Eventually this soccer/rugby style hybrid evolved into American football.
Due to the geographical distance, the game of football in the United States developed pretty independently of the games administrated by the Football Association and the Rugby Football Union (although there was certain cross-insemination by the different football codes). By the late 19th century American Football was already significantly different from both Association football and Rugby Football. So the code of football played on college campuses (and professionally starting from 1892) became completely different than both the Association game and the Rugby game in England.
Association football was introduced to the United States soon after it was codified in England. The first group who tried to codify soccer on a national level in the United States was the American Football Association (AFA) in 1884, the AFA created a national cup known as the American Cup that was similar to the FA cup in 1885. Due to conflicts in the AFA, the American Amateur Football Association (AAFA) was created in 1911 and they created their own cup in 1912. Both organizations applied to join FIFA, but eventually the AAFA won and became the organization known today as US Soccer.
In the 1920s there were a variety of different soccer leagues in the United States, but a true national organization and league system like the Football League in England was never created. Various leagues vied for dominance, but when the great depression hit in 1929 most of them were no longer viable and folded. After the great depression, soccer therefore only survived in the United States as an amateur sport.
In the 1960s, there was an attempt to introduce soccer as a major professional sport again. This led to the creation of the North American Soccer League (the NASL) which was hugely popular for a time. Led by the New York Cosmos, the NASL threw plenty of money at big stars such as Pele and Beckenbauer that allowed the teams to draw tens of thousands of fans per game. However, due to mismanagement and overexpansion, the NASL collapsed in 1985, leading to the demise of professional soccer in the US and Canada.
In 1994 FIFA hosted the World Cup in the United States, leading to another huge surge of soccer's popularity in the United States. This led to the creation of Major League Soccer (MLS), but again, due to overexpansion and mismanagement in the MLS saw quite a few teams collapse within years of its establishment. Today MLS is the top tier league in the United States and Canada, and is growing again steadily, however they have certain controversial rules that differentiate it from other soccer leagues like a salary cap and a draft designed to prevent over-expansion and financial mismanagement.
So in conclusion, Association Football was never that popular in the US due to another code of football was developed independently and became more popular (American Football). The game was also fractured and mismanaged, with poor management dooming many of its leagues. These factors combined contributes to the poor popularity of Association Football in the United States.
Edit: Sources
https://www.ussoccer.com/about/history/timeline http://www.thepeoplehistory.com/footballhistory.html http://www.profootballresearchers.com/articles/No_Christian_End.pdf https://qz.com/206259/a-brief-history-of-soccer-in-the-us-and-why-it-might-finally-have-found-its-place-in-the-american-psyche/