It's hard to generalize since there are 50 states and each state's history is unique, but in the case of the eastern states there were often two compelling factors in the Early Republic. The first was after the American experience in the revolutionary war and war of 1812 it was realized that seaside capitals were very vulnerable to attack from European navies, so moving to the interior was good for a state security standpoint (and in those early days states played a huge role in war efforts). The second was there is often tension between easterners and westerners within the states, and moving the capital to a more centralized location (as in the case of Pennsylvania) was good for compromise.

There was a definite strain in 19th-century American political thought that putting the state capital outside of a major metropolis would help prevent the capture of state government by urban political machines -- although in practice this tended not only delay the process by a few decades.