This is more of a question for /r/AskScience, since it pertains more to Time Lord physics, but as a scholar of the modern history of the Whoniverse, I'll attempt to answer.

It's my understanding (as a historian, mind you, not a scientist) that looking into the heart of a T.A.R.D.I.S. can be a traumatic experience. While all young Time Lords were forced to look into Untempered Schism, which is a rift where one can gaze into the Time Vortex, not all Time Lords get through the experience unscathed. Some are driven mad by the experience, as seen by the Time Lord known as "The Master."

Furthermore, as the Eighth incarnation of the Time Lord known as "The Doctor" has shown, taking the power of the Time Vortex at the heart of a T.A.R.D.I.S. into oneself can cause acute mania and a nigh-unquenchable drive for destruction. Time Lords utilizing the power of the Time Vortex may have become as the Daleks themselves, and could very well have ended up destroying Gallifrey in such a manner.