Incarceration is a relatively modern phenomenon. There are still nations that engage in public physical punishments. Punitive incarceration became the standard in the industrialized world in the 20th century. During the 20th century, various nations engaged in mass incarceration. The Soviet gulags had an estimated 2 million prisoners at their peak, which would give 1950 Russia much higher incarceration rate than the contemporary United States. Apartheid South Africa had very high incarceration rates. Russia continues to have very high incarceration rates, although those have gone down in the last decade.

Through the 1960s, parts of the USA still practiced public executions and physical punishments. Lynching was common across the southern states. In the west, lynching was used as was corporal punishment. I worked with an old fellow in the late 90s in Idaho who had his thumbs cut off in his youth for cattle rustling in the 1950s.

Starting after the Civil War, the south saw a large increase in incarceration rates, particularly for Black Americans. Prison labor was leased by the state to wealthy plantation owners and mine owners as a legal replacement the labor that had been done by enslaved people. Prison labor was curtailed in the 1930s when it was made illegal to transport prison made goods for sale across state lines, however, a temporary law was passed legalizing the sale of prisoner made goods in 1979. This law was made permanent in the 1990s. Today, Alabama, Arkansas, Texas, and Georgia do not pay prisoners at all for their labor, and federal prisons pay a minimum of 23 cents and a maximum of $1.15 per hour.

The Nixon administration started the War on Drugs in 1971. John Ehrlichman has admitted that the war on drugs was a part of a broader culture war designed to disrupt the black power and hippy movements. The Drug War resulted in a dramatic increase in the cost of street drugs. Simultaneously, Nixon opened trade with China which decimated American industry. American industrial cities lost the careers that less educated people could easily get. Property crime increased sharply through the 1970s. Violent crime also rose but at a much slower rate.

Mass incarceration became an American phenomenon under Ronald Reagan. The penalties for drug crimes expanded dramatically with bipartisan support. Penalties for all drug crimes were dramatically increased at the state and federal level. Budgets for drug enforcement rose dramatically. As the penalties increased, so did the profits. Simultaneously, the budget for housing and social safety was slashed, resulting in a dramatic increase in inner city poverty and income inequality. The combination of high poverty, low employment opportunities, and high drug profits strongly correlated with extremely high murder rates. The only other comparable spike in homicide rates in America occurred during Prohibition.

By the early 90s, we have an extremely high prison population and extremely high rates of homicide. Inner city Democrats pushed for increased police budgets. Under Bill Clinton, police budgets dramatically increased and most departments militarized some units. Also, welfare reform pushed more people into extreme poverty particularly in inner cities. The notion of recidivism or career criminality became popular. Studies do show that being incarcerated increases your chances of being incarcerated again. Little attention was given to rehabilitation that worked, instead the prevailing notion was to make the streets safer by permanently removing these dangerous individuals from the streets. States with large prison populations enacted three strikes laws, giving life sentences to anyone convicted of three felonies. Mandatory minimum sentences were dramatically increased. Selling a small amount of marijuana while in possession of a firearm could result in a life sentence if you had the right criminal record and lived in a place like Texas.

By the early 2000s, murder rates had fallen sharply, but incarceration rates continued to climb. By this time, we had a large prison industry, where private companies were leasing prison labor and highly budgeted police departments. The people most likely to be incarcerated are poor, particularly poor people of color.

In Europe in the 1970s, a different sort of ideology was forming. Norwegian Thomas Mathieson argued for the abolition of prisons as early as 1974. Europeans increasingly turned their prisons into institutions of reform rather than punishment. Scandinavian prisons have incredibly low rates of recidivism. They are less likely to criminalize behaviors, e.g. sex work. They are far more likely to provide mental health services and basic needs such as food and housing. The other major shift elsewhere is community restoration exemplified by Japanese criminal justice. That idea focuses on restoration and restitution instead of retribution and incarceration. Community restoration is more mediated than adjudicated.

It's hard to compare violence in industrialized nations because what constitutes criminal violence varies dramatically from nation to nation. A fistfight in France is usually not a criminal offense. The only easily comparable violent crime is homicide and we far outpace modern industrial nations because the ease of access to firearms, particularly handguns, makes it far easier to commit homicide in America.

It is unlikely that we are going to see a shift away from large scale punitive incarceration in the immediate future. Crime, particularly homicide, is on the rise again. There are very few politicians talking about economic inequality which is the factor most strongly correlated with crime. Instead, the rhetoric is around increased law enforcement. The defund the police movement was never adopted as a policy, but it is proving to be a convenient scapegoat for political means. Angela Davis has preached prison abolition for as long as I can remember, but it also remains a radical ideology.

Sources (partial)

Foucault, Michel (1975) Discipline and Punish: The Birth of the Prison

DuVerney, Ava (2016) 13th

FBI Uniform Crime Report, https://www.fbi.gov/how-we-can-help-you/need-an-fbi-service-or-more-information/ucr

Bureau of Justice, https://bjs.ojp.gov/

https://harpers.org/archive/2016/04/legalize-it-all/

Mathieson, Thomas 1974 The Politics of Abolition

Killer Mike (2012) Reagan

While there's more to be said, this answer by /u/kylet357 and some replies to that might give you some pointers.