It was originally derived from the Irish term toraidhe, meaning highwayman or bandit, and then applied to supporters of Catholic Duke of York (later James II) in his succession to the throne of England. The political party known as the Tories were originally formed of Yorkist Tories (that is, supporters of James II) around 1680. Around 1830, they changed their name officially to Conservatives, but the nickname stuck around.

EDIT: The Irish word was toraidhe, derived from tóir, my bad.