TechnoGiraffe gives a pretty incomplete answer here.

The Jews were disliked by more than one ancient Empire for two reasons. First, they refused to grant deified status to anyone (thou shalt have no other gods before me...), AND they were in general very very difficult to control, impossible to assimilate, and in general made themselves 'different' and separate almost everywhere they went. For the Romans, this was of course a problem. The revolt of 70/71 was a big one, and resulted in the destruction of the Second Temple and the plundering of Jerusalem's (really, the aristocratic and religious castes') treasure back to Rome (depicted here on the Arch of Titus: http://upload.wikimedia.org/wikipedia/commons/7/7d/Arch_of_Titus_Menorah.png). However, this was not the death knell for Jews in the Levant. To the North and East of Jerusalem relative independence remained for quite a while, although the area from the coast, inland to the city was devastated. However, the destruction of the Second Temple ended once and for all the role of Jerusalem as a central place of worship, and started the Jewish (and later Christian) traditions of setting up worship in smaller groups centred around the books rather than the place.

Fifty years later, the Emperor Hadrian had the great idea to rebuild Jerusalem, with a temple of Jupiter replacing the Second Temple (as a 'favour' to the Jews...silly billy). Needless to say, the remaining Jews were pretty upset by this, and the Bar Kokhba revolt began. The end result was a depopulated Israel and a scattering of the Jewish population. Hadrian renamed Judea to Palestina and supposedly banned Jews from the place. Of course, many remained, and even at this point there were large Jewish communities in many large cities across the Empire (separate and not assimilating of course, as mentioned above). There were some later migrations in and out of the area (affected by Persian, Arab, and Crusader invasions) to the point that it is very doubtful more than a community or two has survived in Palestine over this whole period, the rest being more recent immigrants (before and after the Balfour declaration and the formation of the state of Israel).