Even the smallest county in Georgia has this building: https://en.wikipedia.org/wiki/Taliaferro_County_Courthouse
Most of the counties I drive through have very nice courthouses that seemingly would cost a lot of money for such small populations. Is there a particular reason for this?
Yes, this was common in many states in the early history of the United States, but certainly has a particular association with the South where the looming courthouse was an enduring image well into the 20th century. It was the seat of county power - wielded by the 'courthouse clique' - and of county society as well. To quote Charles Reagan Wilson's summation "The courthouse square is perhaps the most potent, distinctive symbol of southern community." It is important to keep in mind that for those rural counties, the county seat was going to be one of the few large settlements in the entire county, and this was the center of the town for an entire county essentially. Although the centrality was true in the North as well in the earliest days of the country, the development of urban polities in the North, and declining importance of community censure that the courthouse represented, meant it didn't hold quite the place of pride for Northern communities as their Southern counterparts.
When court was in session, it was a social occasion which would bring interested persons (by which I mean white men) all into town to, some who had issues before the bench, but others simply there for the chatter, gossip, fights, and the drama of the courts themselves - no small matter in a society so overly concerned with enforcement of behavioral norms and punishment of those who transgressed social expectations. Although Wyatt-Brown doesn't put it in quite so explicit terms, it isn't hard to see in his description of an ante-bellum session on Davidson County, North Carolina as providing something of a soap opera for those looking for entertainment in the pre-television era. Court days were not the only part it played though. In most towns the market would be held on the courthouse square, voting would be held there, the periodic militia musters would be held in front, in the post-war period monuments to the 'Lost Cause" were erected, and in the antebellum period, slave auctions would be held in front, if not literally on the courthouse steps. Its central role then of course meant that storefronts around the square were the hottest real estate in town, resulting in other important businesses being in close proximity as well.
So anyways, the point I think is clear enough here, namely that with the courthouse being such an important and central part of county life, investing in a grand imposing courthouse was a matter of civic pride. They wanted a Courthouse that literally towered over the community in the same way that it did figuratively. During the early expansionary period, when a new polis was created a log cabin courthouse would be just about the first thing erected simple because it was one of the most important buildings in the mind of the settlers - and in due course it would of course be rebuilt to be much grander. We also see the creation of new counties in existing states. In the post-war period, South Carolina for instance carved out 13 new counties between 1882 and 1919. One of the core reasons being to reduce the size of the counties and allow for a 'more central courthouse' that is easier to access for the residents. A community needed a courthouse to feel like, well, a community, and no Southern country, however small, felt complete without a nice one.
IF you can't get an answer, try /r/askarchitects/