Because it wasn't just the Colosseum, in fact, it would have been pretty far down the list of buildings to be stripped because it was used as a meeting are and Christian center of worship. Rather, stone and brick was taken from all the other countless structures that remained--Rome was a massive city, far larger than it would be again until the nineteenth century, and enormous quantities of stone and brick were used all across it. Just look at the Forum today compared to a reconstruction, and relatively speaking it is in excellent shape.

The first chapter of Stephen Dyson's Rome: A Living Portrait of an Ancient City has a good discussion of this issue.