It will depend entirely on what you're trying to research, how long ago it happened and what kind of history you're trying to write. But broadly speaking, you've touched on the main ways above. Historians use records held in state and private archival collections (the biggest of which are so massive that no single individual could ever hope to view more than a tiny fraction of what they contain in a lifetime), they use published material found in libraries or available to purchase or, if the topic is otherwise recent or obscure, they can try and gather sources themselves, conducting interviews or searching for documents held by family members. Physical evidence from archaeological work might also be useful, depending on the study. Pretty much anything produced in the past can be a primary source, so they can, in turn, exist just about anywhere.

Where I see the crux of your question is more about how historians go about deciding where to look for such material, given that it could, in theory, be anywhere. The answer here vary again depending on the aims and nature of the work, but it's possible to give an overview.

Documents considered 'important' either at the time or since - government papers, political speeches, major newspapers and so on - tend to be relatively well-catalogued. If you, say, wanted to know what the State Department thought about Japan in the 1950s, it is easy enough to find out where State Department records are held, and the combination of professional archivist assistance and cataloguing would make it relatively straightforward to start looking at relevant documents. As your expertise and familiarity grew with these records, it becomes less random and more targeted as you work out where the things that interest you are kept and what the 'logic' of the archiving system is. You can also generally make use of the work of other scholars before you, since you aren't the first person to be interested in such a question - their footnotes and bibliographies will give a sense of where relevant material lies. You generally don't want to be retracing their footsteps perfectly - presumably you aren't just trying to rewrite their work! - but it can be enough to give you a launching off point.

Speaking more specifically about 'today's historians', most major archives will have a lot of metadata about their collections available online, in the form of digital catalogues, finding aids and access to archivists, so it's possible to do a lot of this groundwork before even visiting an archive. The widespread digitisation of primary material over the past 20 years has also made it much easier to access certain sources, though it's rare that everything you'd need for a large project would be available digitally. For more obscure topics for which the archival collections are more scattered and less well-resourced, the likelihood that much of your work can be done remotely dwindles - in some cases, the archivists themselves might have only a loose idea of the material they hold that's relevant to your project. In some cases, it might be conservatism or neglect rather than lack of resources - even governments sometimes hold limited interest in making their archives accessible, and so research becomes more like a fishing expedition, trying to deduce the location of relevant material and searching for it by hand. This is obviously more time-consuming, but holds the promise of more original discoveries.

If you're looking to create your own source base, this becomes a question of making interpersonal connections and building a network of contacts who can hook you up with privately-held source material or people to interview. The work can be rather adjacent to genealogy, and might indeed make use of ('real' or digital) communities of people interested in their family or local histories. Again, this can involve a huge amount of work, but can pay off in terms of access to completely fresh perspectives and source material. Ultimately though, it revolves around your ability to get people to buy into your project and its aims, and your ability to walk a tightrope between producing critical and insightful histories while maintaining the goodwill of the people who made that possible.