How were ancient astronomers able to discover planets without telescopes? When did it first come up and how?

The planets known before the identification of Uranus as a planet in the 18th century are very bright. With the exception of Mercury, they're very prominent in the night sky (Mercury is bright on average, and can be very bright, but it's also close to the sun in the sky, and can be difficult to see). If they were "fixed stars", i.e., stars, they would have been well known to ancient astronomers simply on account of their brightness. As planets, from the Greek πλανήτης (planḗtēs), meaning "wanderer", they attracted further attention due to their motion about the sky. Their brightness meant that they were known, and their behaviour in the sky - they move relative to the fixed stars - made them special, and identifiable as a category (their brightness also varies).

The planets were not the only moving objects in the night sky. There were also meteors and comets. But these were transient visitors (and in the case of meteors, very transient). The planets were bright objects in the sky that moved relative to the fixed stars with a consistent and predictably motion. Note that taking this last as a definition of "planet", we would include the Sun and the Moon as "planets", as was indeed ancient practice in western Asia, the Mediterranean, and India (even then, the Sun and Moon were recognised as especially special, and the classification would often be Sun, Moon, and the other 5 planets). Venus, Jupiter, Mars, Mercury and Saturn were recognised as planets by the major astronomical systems of Eurasia that left us plentiful written records (western Asia, Mediterranean, India, East Asia).

Just how bright are the planets? The usual description of brightness is the "magnitude". Originating from a classification of stars into 6 categories, with stars of the first magnitude being brightest, the 2nd magnitude being fainter, down to the barely visible 6th magnitude stars, this became a logarithmic (to base approximately 2.5) quantitative scale. The smaller the number, the brighter the object. Negative magnitudes are the brightest. The magnitudes of the naked-eye visible planets and the 4 brightest stars in the night sky (and the Sun and Moon) are:

Object	Magnitude
Sun	-27
Moon	-13 (when full)
Venus	-4.1 (mean; varies from about -3 to -5)
Jupiter	-2.2 (mean; varies from about -1.7 to -3)
Sirius	-1.5 (brightest star)
Canopus	-0.72 (2nd brightest star)
Alpha Centauri	-0.27 (3rd brightest star, combined brightness of binary system)
Arcturus	-0.04 (4th brightest star)
Mercury	0.23 (mean; varies from about +7 to -2.5)
Saturn	0.46 (mean; varies from about +1.2 to -0.6)
Mars	0.71 (mean; varies from about +1.9 to -3)
Uranus	5.7 (mean)

Note that the ancient planets are, at their brightest, among the brightest objects in the night sky. When it is at its brightest, there are only 3 stars in the night sky brighter than Saturn, and the other ancient planets are, at their brightest, brighter than any of the stars in the night sky. There are only 4 stars in the night sky brighter than magnitude 0 (i.e., with negative magnitudes); these 4 stars are listed in the table above. There are only 15 stars brighter than magnitude 1.0; even at their mean brightnesses rather than their maximum brightnesses, the planets outshine almost all stars in the night sky.

It's difficult to miss them!

It's interesting to note that the 1st planet discovered in modern times, Uranus (discovered by William Herschelin 1781) is visible to the naked eye (with good seeing - don't expect to see it with city light pollution). However, at a magnitude of 5.7, it isn't bright, and doesn't attract the attention of the brighter planets. There are almost 5000 stars brighter than magnitude 6.0, and it just wasn't noticed that it was a planet.