There has been an enormous amount written on this topic. Certainly people like Pierre Payer, R.I. Moore and others have offered some interesting observations. Moore famously argued that Fourth Lateran marked the beginning what he termed a "persecuting society" in which religious heterodoxy, confessional others, behaviour dissidence, and sexual deviance all became increasingly targeted by institutional mechanisms and organizations that sought to produce a more homogenous and pure vision of Latin Christendom.

For people like Moore the idea of forcing everyone to confess once a year was connected to a change in the Church's approach to inquisition and the policing of heterodoxy. From the more ad hoc nature of episcopal inquisition seen in the twelfth century we begin to see the development of the papal inquisition in the beginning of the thirteenth which happened to coincide chronologically with the apogee of papal temporal power in the medieval world under Innocent III. Moreover, this period saw not the emergence but rather an increase in awareness of certain "heretical" communities throughout Latin Christendom, particularly in northern Italy, southern France and elsewhere - the Cathars being the most obvious example.

While heresy had always been a feature of Latin Christian Europe it had been relatively quiet, with sporadic outbreaks in Orleans and Montreforte in the early eleventh century and a few outbreaks in the twelfth century such as those led by Peter of Bruys, Henry of Lausanne, and Tanchelm. The later twelfth century and early thirteenth saw a significant expansion in the number of perceived threats to orthodoxy (consult scholars like Malcolm Lambert if you would like to learn more about this topic). Whether this meant that these communities rapidly expanded, such as the Waldensians, or simply became more visible and concerning to Latin Christian authorities is open to speculation.

These processes, along with the advent of the great preaching orders, the Franciscans and Dominicans, dedicated to the evangelization of the Catholic faith and the confrontation of heresy and the truth claims of other Abrahamic faiths, contributed to an increased willingness on the part of the Church and its orders to play a more prominent role in the shaping of public order and religious belief.

The canons of Fourth Lateran and the formal demand for all Christians of both sexes to attend confession once a year can be seen as another feature of these processes in thirteenth century Catholicism. Forcing people to go to confession allowed the Church to, as it were, place the inquisition in every parish church, and recruit every parish priest into the business of spiritual direction. Every Christian had to submit themselves to this form of self-reflection, cleansing and penance, which helped to police heterodoxy and dissident behaviour at the local level and bind people more firmly to the power of the institutional Church.

This is a very interesting topic and there are many scholars who write on the study of confession and penance who don't know the exact origin of confession and confession manuals which become prominent in this era (Eamon Duffy, Leonard Boyle, and Miri Ruben are among the most well know scholars who don't have a precise idea; Robert Grosseteste, William of Pagula and William Dunbar being a few of the medieval theologians and clerics who wrote confession manuals). Joseph Goering suggests that oral confession was not common practice before the 13th century, but that Innocent III did a number in institutionalising it to the public at the Lateran Council.

The church at the time began investing a lot of time in the souls of the people of the parish at the time, beginning to understand that the world and everyone who lived in it were actually created by god (rather than a world unworthy of God) and everyone was worthy of being saved by god and for heaven (rather than bringing the sin and devastation/destruction of evil upon them and the world). Until this time, most literature and written work associated with the church was written for the church officials, theologians, and monks, not the general population. The people weren't very aware of what entailed "sinning" but, as R. Emmet McLaughlin suggests, the church was clearly at this point investing in the effort to bring penance to the parish, so the confession manuals were written for the penitents/laity (in their languages, rather than in Latin) to cleanse them and prepare them for the afterlife.

The first action was to record the community's wrongdoings in the visitation records by the bishop, but these do little for scholars in capturing the whole picture of the laity and penance as the visitations were only anually and focused on all things, from penance to upkeep of the property. The shift to prescription of penance from bishops to priests (local to the area and permanently located) was an easy one.

However, in order for the people to be cleansed through penance, the priests needed to know what the person had done wrong to be able to prescribe penance. The concept behind all this was that the confessor was morally sick, in a way, penance was the cure, and the priests were the doctors who were educated in the field to prescribe these things. Oral confession developed from this need to know how to fix the everyday person who didn't know what was considered a sin, and was actually more so done in public than in private booths that we associate with modern catholic confession (didn't develop until a few centuries later around the Council of Trent).

After reading u/trinity- comment, the persecuting society reason is definitely an argument for understanding the institutionalisation of confession under the fourth Lateran Council, but it developed on an individual level originally, less about heresy and more about saving the individual soul, before expanding into its institutionalised function against heresy.