There is a book that came out not too long about the tenth of a second by Jimena Canales, which delves into the scientific bickering which led to the definition of, well, a tenth of a second. Though I feel Barbara Adam's Time is an excellent introduction into humanity's attempts at conceptualizing time, in which she goes from pretty much our earliest mythologies to the present day to track exactly how humans have tried to measure time and use this knowledge to organize their lives. Her work isn't necessarily focused on the length of a second though. I'll try to answer your question but it's honestly quite a rabbit hole that could possibly extend back to our earliest maths.

When it comes to time, what humans have basically always done is look for stuff that happens regularly and use those repeating cycles as their basis for social organization, which in practice meant looking at the sun, moon and star cycles and basing their time on that. To be clear, the historiography has moved past classic conceptions of cyclical time perception having been replaced by a linear time with the rise of modernity. Instead you could see the planet's motion through the universe as humanity's first clock, as something that repeated with a reasonable expectation of uniformity. Throughout human civilizations there are divisions coinciding with the solar and lunar cycles, and in a sense you could trace a direct line of inheritance from ancient Babylonian months to our own current system. There is in fact a whole discipline called archaeoastronomy which attempts to read prehistoric monuments as astronomical calendars.

Now what does this have to do with the second? Once you chart the planet's rotation around the sun as taking approximately 360 days, a circle having 360 degrees suddenly becomes very useful for astronomic calculations. I've recently seen in a video that Sumerian mathematics used a sexagesimal number system, meaning it's based on the number 60, which would also help explain why they ended up dividing the day into specifically 24 parts which can be cleanly divided by 12, 6, 4, 3 and 2. I'm certainly no historian of ancient Sumerian or Chinese mathematics so forgive any errors, but the Chinese also divided their days into similar divisions (I believe also of 12 but I would need to check to be sure). We kind of ended up inheriting those divisions of time as time went on, having 12 hours of daylight and 12 hours of nighttime alternate daily as the standard in Judeo-Christian time.

We're still not there yet. These hours weren't uniform, they were still mostly based on the sun's cycle meaning they became longer or shorter depending on the season. The final step was for the mechanical clock to replace the earth's rotation as the ever-repeating pattern in which we could base our social organization. This is a tricky subject to tackle and I won't pretend to have the final answer on this transformation, though Vanessa Ogle's Global Transformation of Time attempts to tackle the question head-on. Suffice it to say various historians have wildly differing opinions on the nature of this transformation, though there is a consensus that the clock helped set in motion some kind of change which in turn became intrinsically linked to our passage into our still prevalent conception of time.

Part of this transformation, which to be clear needs to be situated between 1600-1900, meant the 24 hours of day night became uniform, abstracted and no longer tied to the seasons. Astronomers before that point had already used degrees and thus minutes and seconds in their calculations, which all points back to that original circle of the Earth's rotation and which eventually became co-opted by states worldwide as the single time standard. This in effect meant the length of a second had also been determined, considering the 24 hours were still based on the annual cycle of the Earth and could thus be astronomically calculated down to the minute and second.

By the way, today our second isn't defined this way, but instead is based on the amount of oscillations of a Caesium atom in certain circumstances as a way to link our conception of time with more fundamental cycles within the universe. Now to be clear, I've rushed through history all the way from prehistory to now and I've glossed over a lot of things, including different calendar reforms over the years and how certain elements of this new calendar seemed to catch on quicker than others. I also mostly told it from a eurocentric perspective as this is where scholarship tends to place this transformation, and it was afterwards exported throughout the world through their imperialism (even by states who weren't necessarily subject to it directly). It's an interesting subject though, and I hope to have somewhat answered your question.