What are some trustworthy books/references on the Freemasons and Freemasonry?
I am interested in the way they (likely have) influenced history, the symbolism, rituals and the grade system, but without the conspiracy nonsense.
So, you have already highlighted the most obvious problem when it comes to reading about Freemasonry. There is a huge swathe of conspiratorial literature out there from the politically motivated 'Elders of Zion' segment right through to the outright biased axe-to-grind cohort such as Stephen Knight's The Brotherhood. I am sure you are well aware of the issues with these texts and I won't dwell on them further.
The other problem lies at the feet of active Freemasons and/or sympathetic writers. Unfortunately, many 'big up' the nature of Freemasonry and, as you have stated in your question, their influence on history. As a small case study, I addressed this very topic in a previous question regarding the formation of Latin American states. Another element of this is masonic authors wanting to trace the lineage of Freemasonry back to ancient times with very little concrete, and rigorous, academic analysis. This is best exemplified by Robert Lomas' controversial The Hiram Key. Be careful as well with any material that claims a consistent link between modern Freemasonry and the Templars. The Knights Templar is a particularly interesting topic but I recommend Helen Nicholson's book on the subject for historical accuracy.
Note that when it comes to the discussion of rituals and ceremonies, it is often from a very personal angle when an author discusses the meanings behind each. Masonry is designed to provide a path to discovery rather than laying out the 'answers' in plain sight, it is something to bear in mind.
So, between these two bulwarks of 'bad history', where is best to look? Ultimately, I encourage you to rely on sources written by academics or ones that go through some sort of peer review process.
RF Gould's Concise History of Freemasonry is often cited as a go-to for this sort of question but it has to be caveated. This was a work published in 1903 and is regarded as, quite honestly, dry. It is intricate in its detail and labours in its narrative feel. It will certainly give you a good grasp of the fraternity but is not the most 'exciting'. In a similar vein, AE Waite's New Encyclopedia of Freemasonry was published in 1921. It is solid in its storytelling but clearly shows its age. It should be noted, that although not wholly apparent in this particular work, Waite was a dedicated hermeticist and mystic and approaches topics in that mindset.
One of the best resources for masonic history would be the Ars Quatuor Coronatorum, an annual journal published by the Quatuor Coranati lodge which is dedicated to the pursuit of masonic research and is home to many academics in this field. I would also add the Bristol Masonic Society and the Manchester Masonic Research Centre to the list of trusted sources.
If you require a more thorough list, and scans of any of their publications please send me a message/reply below. This may or may not be applicable too, but if you have an interest in joining a masonic lodge, try not to ruin it by delving too much into the ritualistic element.