This is my stock holier-than-thou answer to questions like these (fair warning; I get pretty damned holier-than-thou):

Ideally, one should go to university to get an education, not to get a job. So if you're interested in being educated in history, I say that that would be reason enough. However, there are other considerations. What can you do with a minor in history? You can develop major research skills and writing skills. You can exercise your curiosity, imagination and critical thinking. You can learn how to ask the kinds of questions that generate interesting answers, and you can learn how to find those answers.

I work in the Humanities because they help me to expand my understanding of what it means to be human.

(Although, on a semi-practical note, all of the skills I've listed, and probably more, are soft skills that you can apply in your work in Physics and pretty much whatever job you get in whatever field you end up in. So yeah, totally worth it.)

Why not? Practically, you'll learn to read critically and you'll learn to think through how a number of causes or factors that shape an event (physics will teach you that and history will help reinforce it). While scientific writing has some key differences from history writing, you'll almost certainly improve your writing skills. Clear and readable writing is useful in the sciences. History of science classes might also help you understand some of the background of the science in your major.

More generally, between work, personal obligations, and that pesky need to sleep (which has frustratingly increased as I've gotten older), often there isn't time to keep up with things that I find interesting, but aren't directly related to work. If history interests you, take advantage of the opportunity to engage with it while you have the time and energy.