Clock coordination was a major issue in the 19th century, not because of pocket watches, but because of train schedules. Get those wrong and not only can people miss their trains, but if trains that share track don't know what time it is fairly precisely... disaster!

There were many ways to do centralized time coordination but they all worked on basically the same principle. You have a "master clock" that is setting the time for your town or principality. Other clocks set their time based on the master clock. This delegation of time is pretty straightforward: you can use electrical signals, you can use pneumatic signals, later you can use radio signals. (This is how a consumer "atomic clock" you buy today works: it's just a radio clock that is being set by a signal from a centralized authority. Your computer is likely getting its time from a central authority as well.)

OK, so how does the master clock know what time it is, really? You ultimately need some increasingly centralized "authority" to delegate the time. This role was initially served by astronomical observatories. The most famous of these is the Royal Observatory in Greenwich, for which Greenwich Mean Time is name. Basically the Observatory would say, "OK, it's noon exactly... now!" when the Sun was at its zenith. And that would then be used to coordinate subsequent clocks, either manually or through the means I've described.

It was during this period that time zones became standardized as a way to prevent any given location's "local noon" solar time from complicating coordination. Railroads initially began coordinating time between stations (so the station clock would be authoritative railroad time), using the means outlined above.

So you, a random citizen, would be setting your clock based on another clock — perhaps the large ones at the railway stations, which you knew were ultimately coordinated back to some centralized authority. It is of some historical interest that one Albert Einstein, at the same time he was thinking his first thought experiments about how you would coordinate clocks without a centralized authority (a key part of his thinking on relativity) was doing so while reading time coordination patents in the Bern patent office where he worked. And on the way to work, he'd walk past multiple buildings with clock towers, letting everyone know what time it was.

In late-19th and early-20th century rail stations that still exist, you can often still see such clock towers and coordinated clocks. My own local train terminal (ca. 1907) in Hoboken, NJ, boasts a very picturesque clock tower that is kept in fairly good working condition. Today we see such things as charming but it is the answer to your riddle of how you'd know what time it was, and also an answer to the question of why it would matter: you don't want to miss your train!

For a very interesting read on the history of 19th-century time-coordination technologies, as well as their relationship to some of the major scientific theories of the day (notably Einstein's and Poincaré's work), see Peter Galison, Einstein's Clocks, Poincaré's Maps: Empires of Time (Norton, 2004).