I'm reading Bertrand Russell, and he uses a lot of Greek characters and formal logic symbols in his text, which makes me very curious to know how writers managed to work things like that into manuscripts before handing them in to publishers if they only had their respective language's typewriters to work with. I poked around on Google, but the only things that come up are people discussing different European countries who use the Latin alphabet and the varied standards in machinery required for necessary accents and diacritics, but most older British and American typewriters I've seen or know of don't really have any extra-standard-English bells and whistles like that incorporated into the machinery.
Did they have special advanced typewriters for authors who work in fields with more niche character needs, did they just handwrite the relevant notes on the pages for their editor, or is there a simpler explanation I'm missing?
The most typical solution was just to write it in by hand. Others have already mentioned this, but I thought I'd offer up some specific examples since I have a number of texts where this is the case.
In linguistics this was a bit of a problem, since you can't really type IPA on a standard typewriter. The solution was generally then to hand-write. I have a couple specific examples I can give to show just how major of a problem this could be.
These are both from a doctoral dissertation from 1969, all done on typewriter, with hand-written modifications. I can't imagine how time consuming this must have been, to go through and painstakingly modify all the places that needed it while otherwise in the process of getting it all written down.
Here's one section that is comparatively minimal. ɦ was written by hand-writing a top hook on h. ŋ was done by adding to n, and ʔ was just written in by hand from the start. Presumably the author was doing this immediately after typing the letter each time.
It get's a little hairier here, adding ŋ and ɤ directly, nasalising ã and ɑ̃. Some of the charts get a little bothersome to read when there's a lot going on.
Then of course there are charts, also basically hand drawn, and let's not forget that this typewriter didn't have Chinese characters, which were also needed.
Then there was also the issue that a lot of things weren't as standardised as they are, so a lot of older linguistics texts were using letters like ɿ or ʅ, things that no typewriter head would have ever had. So even with the best set of Selectric heads, you were still going to have to hand-write things, although those did exist for different scripts or special symbols. Just not for everything people were needing to write.
Sometimes I have to stop and think just how amazing it is what we do today with LaTeX and how good we have it. It's pretty astounding considering how this all used to be done that I can so effortlessly switch between English and Tibetan in a single paragraph and never once have to stop and think if the publisher will have trouble with it.
This gives you a nice feel for how 'galleys' were created for newspapers and then used to create metal plates for printing.
http://commfaculty.fullerton.edu/woverbeck/dtr5.htm
Basically, however, for specialist equations an artist would have drawn the equations by hand or have used rub-down transfers to create the equations that would then have been photographed and inserted into the pages
Basically, they either drew the characters with a pen or used exchangable typewheels or -heads. The IBM Selectric line was famous for having easily exchangable heads with all sorts of weird characters.
into manuscripts before handing them in to publishers
Wouldn't most manuscripts be really manuscript, i.e. handwritten by the author? And if circulated typed on a typewriter, that work would have been done by a professional? (I know that in older papers and books, I've seen thanks to the secretaries which have had to handle the special requests and that in the early 90 when I graduated, some people where still using such kind of services to prepare their memoir while other were typing their work themselves, either on a typewriter or on a computer)