This is a fairly easy question to answer, as Bible questions go: the Biblical hostility to Babylon is due to the fact that the Kingdom of Judah was destroyed by the Babylonians in 587 (2 Kings 25, much of the book of Jeremiah). The ruling and literate classes were deported to prevent rebellion.

The northern kingdom of Israel had been destroyed a hundred and thirty years earlier by the Assyrians, against whom a grudge was also maintained (see the Book of Jonah, which depicted a Hebrew prophet who is willing to die under YHWH's condemnation rather than preach to the Assyrians, fearful that they might repent and be spared divine wrath.)

There are definitely parts of Babylonian culture the Hebrews disliked (astrology gets singled out in a few places, most directly in Isaiah 47:13). The "Tower of Babel" is probably a reference to the Ziggurats ancient Mesopotamians built to access their gods... they were in fact viewed as stairways to the heavens, though the intent in their construction was piety, not vainglory. But the main qualm the writers of the Bible had with the Babylonians was not their magic or their pagan religion--these were qualities of plenty of other nearby civilizations. The real issue was the history of oppression that had been dealt out at their hands.

For an instructive contrast, see how the Persians are described in the Hebrew Bible. They differed from the Hebrews in many important ways, but their policy of cultural and religious toleration (at least as far as the Jews were concerned) made the Biblical authors look on them generally kindly. Most dramatically, see Isaiah 45, where a Persian King is called God's messiah. (Where they are criticized, it is as being overly absolutist and decadent--as in the books of Esther and Daniel.)