As you noted, the destruction of the second temple is a pretty historically verifiable event. It's attested in historical sources from the period, particularly Josephus, as you noted, but it's also manifestly evident in other Jewish sources (and I'm pretty sure Roman ones, too, but I don't know them as well). The only lore-y part is the date of its destruction on Tisha B'av, which is first mentioned in the Mishnah (2nd century). It puts a bunch of bad events at that date, which is a little dodgy--which dates may be harmonized and which may be legitimate isn't clear. I'm unaware of any scholarly discussion of the exact dating of the destruction.

The first temple's greater antiquity makes it a bit murkier. First, though this isn't an airtight historical argument, its existence makes a whole lot of sense. Jerusalem was the Israelite capital, and a major shrine in such a city would be pretty much de rigueur. Obviously excavating such a site would be impractical and extremely dangerous both for archaeologists and for the entire region.

While the bible itself is not a historical text in the sense a history book is, it is a very old text. And it's a very old text that frequently speaks to the existence of a centralized place of worship, even if not all the bible speaks to it as the centralized place of worship. This indicates that at the time various biblical texts were written, some sort of building we'd call the First Temple existed.

To this point, I've never seen a Israelite history person deny the existence of the Temple wholesale. All but the most minimalist dating schemes would put at least one temple-mentioning source to within when it would've stood. To wit, The Bible Unearthed mentions the Temple, and while it demonstrates why the dating of its construction in the bible is likely incorrect, it does not call into question its existence itself.

There is, however, a lot of ancillary lore about the Temple that is not backed up historically. The Talmud relates all sorts of things about it. It talks about the fire on the altar being in the shape of a lion and was unextinguishable, the showbread miraculously being extremely filling (except in times of corruption, when no matter how much one would eat, it would never be filling), the meat of sacrifices not becoming rancid or attracting flies, and there was always enough space for worshipers to bow in the courtyard no matter how crowded it was. There's no hard historical evidence for any of these.