Like, When the first Small Hand-Big Hand clock was made how did they know when to start it up so it wasn't off? How did they know it was Exactly 12:00 or 1:00 when they started it up? Sorry if this sounds stupid, But I seriously have no idea at the moment.
You can tell the exact time of the day (at your parallel) measuring the elevation of the sun in the sky and knowing your latitude and what time of the year is it. However, putting a number on it ends up being a matter of consensus between people, take for example summer saving time on how people can simply decide that 14:00 is now 13:00.