The short answer is that state capitals serve an entire state, and there are intrastate divisions and political motivations that prevent them from being moved to the largest cities.
The longer answer, as with many things involving US states, is that the placement of state capitols is largely dependent on the history and geography of the state in question. For most of US history, the most efficient ways to travel, communicate, and transport goods was by boat; travel by land was relatively difficult, slow, and expensive. So, historically, capitals tended to be in places with accessible rivers that are centrally located, so as to make it accessible to people from all over your state.
You'll notice that a large number of states (ex. South Dakota, Iowa, West Virginia, Arkansas, Ohio, Vermont) have their capitals near the geographic center of the state, but it's more useful to think about where the center of population was for the state when its capital was established.
The question then becomes why didn't capitals move to the largest population centers as travel became easier. And this is where politics enters into things. As previously touched upon, many capitals were chosen to be centrally-located as a compromise between regions or cities, and geographic divides may be keeping the capital from moving, as is the case with Pennsylvania, where there is a precarious balance between Pittsburgh in the west and Philadelphia in the east.
But by far my favorite political reason for keeping capitals away from states' largest cities is that their remoteness is a way to prevent political accountability. A 2014 analysis by Filipe Campante and Quoc-Anh Do (see here) found that people farther away from their capital pay less attention and vote less often in state politics, that newspapers with audiences farther away had less (and less-intense) coverage of state politics, and politicians were more susceptible to outside money.
References:
Campante, Filipe R., and Quoc-Anh Do. 2014. Isolated Capital Cities,Accountability and Corruption: Evidence from US States. AmericanEconomic Review 104(8): 2456-81.
Wilson, James Q.1966. “Corruption: The Shame of the States.” Public Interest, 2: 28-38.
A similar question^[1] was asked regarding the capital of New York State and provides a good answer.
From reading that, one can see that many factors played a part in determining where the capital would be; population was not the sole consideration. Other states will have a similar diversity of reasons for why their capital is where it is and would require 49 more posts of similar size to the one cited for New York State to provide a complete answer to this question.
This question is pretty general, and the answer u/back-that-sass-up gave above does a great job of covering most of these general reasons for state capitals not being in the largest city, but I think a specific example can shed a light on why these factors might have been so important and contentious back at the time that many American states were gaining statehood.
Nebraska's capital is Lincoln: the (now) second-largest city in the state and in the eastern third geographically. While Omaha is nearly twice the size of Lincoln in population and places like Grand Island or Kearney are more centrally located (about 10 years ago the State Fair was moved to Grand Island for this exact reason), at the time of Nebraska's statehood in 1867, there were other factors at play.
Today, Nebraska's center of population is still in the eastern part of the state about an hour drive northwest of Lincoln^1 , but in 1867 it was even further east. The state's population at that time was dominated by two cities: Omaha north of the Platte River, and Nebraska City south of it. While most of the population lived south of the Platte, Omaha was both the largest city and the territorial capital. The first Secretary of the Nebraska Territory, Thomas B. Cuming, who was a member of the Omaha Land Company, gerrymandered the legislative districts to favor the north in spite of the fact that nearly twice as many people lived south of the Platte^2 . This unfair apportionment led to members of the Fourth Territorial Legislature, after "the brandishing of dirks and threats of shooting", attempting to establish a rival legislature in Florence, which was not recognized by the territorial governor, William Richardson^3 .
Although the territorial capital would remain in Omaha, tensions would remain. With statehood imminent, dissatisfaction with Omaha as the capital led to a push to ensure that the future state capital would not be Omaha. Omaha lawmakers, knowing this, attempted to play on still-raw Civil War emotions to keep the capital in their town. Although slavery had never been permitted in the Nebraska Territory, following the Civil War many Southerners immigrated to southern Nebraska^4 and Omaha Senator J. N. H. Patrick pushed through a bill moving that the future state capital would be renamed to "Lincoln" in honor of the recently assassinated President^5 . This bill was intended to ensure that southern Nebraska towns would reject any offer to become the state capital so that Omaha (which would've been renamed Lincoln) could remain as such. However, this tactic ultimately failed.
Beginning in 1866, the Capitol Commission, was tasked with determining the site of the soon-to-be state capital south of the Platte River. Ultimately, the Commission decided on the village of Lancaster, which had been established in 1856 in the hopes of extracting salt from the nearby salt marshes. At the time, Lancaster was considered very centrally located: the Commission reported that "the location was at the center of a circle of about one hundred and ten miles in diameter, along or near the circumference of which are the Kansas state line, directly south, and the important towns of Pawnee City, Nebraska City, Plattsmouth, Omaha, Fremont, and Columbus"^5 .
The choice of Lancaster was met with criticism from some, primarily Omaha lawmakers. At the time of its selection as the future state capital, Lancaster had a population of approximately 30 residents, no railroad, and no river access to the state's major cities. In spite of these drawbacks, Lancaster was renamed to Lincoln and the State Legislature first met there in 1868, one year after Nebraska entered the Union. To get an idea of its remoteness and lack of real settlement, here is an early photograph of Lincoln taken either 1870 or 1871 (https://www.nebraskahistory.org/images/sites/kennard/2849.jpg), the house on the left still stands today, its the oldest remaining structure in Lincoln.
As we can see, the state capital of Nebraska today is located at its present site ("Lincoln" would've been the state capital regardless) because of geographical rivalries between faction on the north and south banks of the Platte River and dissatisfaction with the dominance of the smaller population north of the Platte in the territorial legislatures, and got its name because political differences between those two geographical factions were presumed to be stronger than they actually were. Obviously each state has a different history, but hopefully this illustrates one of the reasons why a state's capital may not be the largest city.
Citations:
1.) US Census Bureau. US State Centers of Population, 2011 https://web.archive.org/web/20140103082820/http://www.census.gov/geo/reference/docs/cenpop2010/CenPop2010_Mean_ST.txt
2.) Potts, J. B. Frontier Solons: Nebraska's Territorial Lawmaker, 1854-1867. Great Plains Quarterly, 1992, 12, 269-285.
3.) Potts, J. B. The Nebraska Capital Controversy, 1855-59. Great Plains Quarterly, 1988, 8, 172-182.
4.) Encyclopedia Britannica Online. Lincoln, Nebraska, United States, https://www.britannica.com/place/Lincoln-Nebraska
5.) Nebraska State Historical Society. More about Nebraska statehood, the location of the capital, and the story of the commissioner's homes, 2000. https://www.nebraskahistory.org/sites/kennard/stathood.htm
Sometimes cities are purpose built to be capitals and sometimes trade hubs and commerce centers grow to become capitals. Other times it's simply the first city established, and war can force a relocation of the government center, too. In some cases all of these happened in the same state! You're gonna get 50 answers for the 50 capitals and most will deal with cities that currently meet your criteria of not being the most populated in their respective state. However a capital being the most populated now does not mean that it was the most populated when it became the capital (and vicey-versey), and one such example I'll focus on - Georgia and her ever moving capital, which has had five official capitals but some claim as many as 17 since 1733 (including war time temporary congressional session locations).
Originally the de facto capital was the first city, Savannah. James Oglethorpe, founder of the proprietary colony, founded that city in 1733. By 1736 he had also founded settlements on St. Simon's Island to the south of Savannah as a buffer to Spanish Florida (which paid off during the War of Jenkins' Ear just a few years later), building both Ft Fredrica and the town of Frederica before adding Ft St Simon's further south on the island. Since he based from Frederica it's generally considered to have been the capital for the next decade (though Savannah was larger), then it was Savannah again and remained so from his departure for England in 1747 until the middle of the Revolutionary War when Georgia faced invasion by the British. In 1778 Savannah was taken and the govt fled to Augusta. Soon it, too, would fall and they hopped around Wilkes Co, even basing from neighboring S.C. for a short time. With the liberation of Augusta they returned, and when the British departed in 1782 they again moved to Savannah briefly but wound up back in Augusta. Many Georgians felt Savannah, in addition to being at the extreme end of the state, wasn't properly representative of the frontier land. In the mid 1780s they decided to build a purpose styled capital city and to model it after Philadelphia. It was named Louisville in honor of the King of France and his contribution to American Independence. It would be placed at the convergence of three major roads linking Augusta, Savannah, and a third long abandoned settlement. Louisville served as the capital until the early 1800s when another purpose built capital city would be voted for, this one named Milledgeville in honor of the sitting Governor of Georgia at that time. Obviously both Louisville and Milledgeville were not the most populated cities at the time of being designated capitals as they largely were settled and built for the purpose.
In the 1830s the rail lines decided to connect the Tennessee River with the Chattahoochee River. As they connected the two a small settlement began at the southern end of the Chattanooga line and lacking a proper name was simply called Terminus. It grew very quickly. By the mid 1840s it had been renamed Marthasville in honor of then Gov Lumpkin's daughter, Martha. It had been connected to the Augusta line and Macon line, allowing a massive transportation hub to emerge. People began to question the name so it was once more changed; the Western and Atlantic Railroad would get the honor of the city - it was named Atlanta. In 1847 the first vote to move the capital happened but was voted in the negative. The War happened and post war a new state constitutional convention was called - but the innkeepers in Milledgeville were said to be refusing any black delegates accommodation, so they moved the convention to Atlanta instead. In 1868 and as a result of the convention it became Georgia's capital at about 21,000 population (vs 28k in Savannah) so it wasn't the biggest city when it became the captial. It wouldn't be until sometime between 1880 and 1890 that Atlanta would overtake Savannah as the largest city in the state, a title held by the city ever since.
So the only time Georgia ever named the most populated city as the capital was the original ly founded city and the brief period pre-war that it served in that capacity. It was a result of the centralized location, emancipation, and the commercial capabilities of a connected city that fueled the growth of the ATL.
Similar question, why is the capital of New York Albany and not NYC?
