In history, its really hard to answer the question of why something wasn't done, without making an assertion that cannot be backed up by some sort of evidence. Evidence this sub requires to not remove posts. If you think about it, this makes sense. If you were writing about a castle you built, do you think you would have a section about why you didn't do something?
The largest stone in Stonehenge weighs 50 tonnes. (1) Moving anything weighing this much is a massive undertaking, even to a site as naturally accessible as Stonehenge. Castles, however, are not sited to be accessible; generally, the opposite is true. Medieval castle doctrine maintained that the best places for castles were places that commanded an area without being easily approachable/surround-able. (2) When you couple these two issues together, it becomes simply inefficient to devote the manpower and man-hours necessary to build a castle out of Stonehenge sized blocks. So instead, smaller, more reasonable blocks were used.
Christopher Marshall, Warfare in the Latin East, 1192-1291 (New York: Cambridge University Press, 1992).