I've read Pliny's Letters and was fascinated with all the small stuff that would consist everyday life at that age. Like how he talked about his job, described the layout of his house, thanked his friend for sending him dates, shared local rumors, described his off-duty activities...
What are some other sources where I can read / listen more about something like that? I tried reading Caesar's Gallic Wars and Xenophon's Anabasis, but they wrote about more epic things, and weren't concerned with describing "boring", everyday fine details that I'm curious about.
As you have noticed, day to day life is not as well reflected in written sources as it might be. Historians of everday life often draw on archaeology to supplement what the sources tell us and what we know commonly has to be cobbled together from many different sources. If you are interested in written material specifically, I can recommend three works, two of them shorter, that provide somewhat humorous insights into the lives of simpler people on the street. They all should be available in translation in English, at least in the Loeb edition.
The first is Theophrastus' Characters, the first work to have been written explicitly about character types. It focusses on ancient Athens in the late 4th century and treats 30 types of excentric characters (the superstitious man, the parsimonious man, etc.) and illustrates their typical behaviour about town. It is not dissimilar to Attic comedy in its observation of typical behaviour and inspired imitators into Modernity.
The second is Lucian's tongue-in-cheek "Conversations among Escort Girls" (hetairai), which is available in the Penguin Classics series translated by Keith Sidwell. It paints a portrait of the trials and tribulations higher class prostitutes might have had to face, or at least how their world might have been imagined from the outside. This is not an undistorted view of ancient every-day life, but a very entertaining read and can lead into the history of women more generally. It is set in a Greek cultural milieu, but was written in the so-called Second Sophistic, a period of great literary refinement during the High Roman Empire.
The third is a bit more arcane, namely Artemidorus' handbook of dream interpretation. Divination allows us access to people's real concerns and how they might approach problems that came up in their lives (stolen possessions, slave ran away, daughter not getting married, health problems, son on military service, fear of having been bewitched, etc.). The work is called the Oneirocritica and can also be found in English translation, but is definitely heavier stuff. It too dates from the High Roman Empire.