Siege Warfare has received surprisingly little scholarly attention so there are not a lot of good academic books on the subject. Most works I've come across are on specific sieges rather than sieges across the entire medieval period. I'm much less qualified to comment on the state of Castle history, unfortunately.

Jim Bradbury's The Medieval Siege is probably your best bet for siege warfare. Bradbury has a very approachable writing style but the book is also quite long (~600 pages if memory serves). It mostly focuses on accounts of individual sieges as well as the development of siege weapons and techniques.

For Castles it might be worth looking at Marc Morris' books on the subject Kings and Castles or Castle: A History of the Buildings that Shaped Medieval Britain. I haven't read either of them but his A Great and Terrible King was a great, approachable biography of Edward I and included some good discussion of the castles Edward built so I'd expect his other books to be pretty good. That said, they'll definitely have a very British focus and so probably not cover everything you might want to know.