Not a historian, but an astronomer, so I might be able to answer this (forgive me if this breaks the rules).

The short answer is the Greek's knowledge of geometry. Unlike stars, planets will move about the night sky just like the moon. In fact, the word planet comes from the Greek for wanderer. Anyways, the Greeks correctly reasoned that, because we could see the motion of the planets, they must be much closer to us. From their studies of geometry, the Greeks knew that things that the motion of things further away appear smaller, and vice versa for things closer. Therefore, they reasoned that the planets must be much closer.

This next part is somewhat conjectural, but I believe that planets weren't discovered to be bodies similar to earth until Galileo, who invented the first telescope and used it to observe the planets.

An accurate distance to the other planets was first obtained by measuring the parallax of Venus during its transit across the sun.