Studied ancient history for year 11 so I can give this a go. I can chase down a friend or two who studied them both if you'd like more info.
Modern is anything from "modern times" (duh :p). So you'll learn about the World Wars, French Revolution, colonisation of Australia, etc. Pretty much anything where society had widespread access to electricity/coal/steam power.
Ancient history is like Egypt, Rome, Greece, and lots of medieval European history.
I would say that its pretty subjective when exactly modern history begins. But in this context I'd guess anything before 1400-1600CE ish is ancient depending on the electives in the subject and who you asked.