Ludlow belonged to the de Lacy family in the 11th century. When the last male heir of the family died, it passed down to two daughters of his until it came to the possession of Geoffrey de Geneville. He, in turn, passed it to his son Peter in 1283. Roger Mortimer, who married a daughter of Peter de Geneville, made use of the castle as a base since he was the leader of a group of barons who dethroned the unpopular Edward II in 1326. The last male Mortimer died in 1425 but Ludlow castle passed to his sister's (Anne de Mortimer) son Richard Plantagenet, Duke of York, who was the leader of the Yorkists in the War of the Roses. It only became part of the Crown estates when Edward IV became king.

Edward of Westminster fled with his mother, Margaret of Anjou when his father Henry VI was captured by the Yorks when he was around 7 years old.

Historical and descriptive sketch of Ludlow Castle by Thomas Wright