Basically, because to make a castle large enough to feed itself is going to require so many men to garrison it (and work the fields) that it won't be able to feed itself, or it will be extremely vulnerable to assault.

These were not the mammoth structures frequently depicted in fantasy films. As a rule, they were compact forts which could be garrisoned by a relative handful of men, and with good reason. The longer the length of wall, the more soldiers it takes to defend it; and, consequently, the fewer troops you have available for your field army. As such, medieval castles were built to economize on manpower. Even large castles might well be something like 550 feet by 150 feet - that's a little under two acres, less than the amount needed to feed a single family with medieval agricultural techniques.