Two main systems were available. The so-called Milesian or "alphabetic" system is the more common and better-known one, and was inherited in a slightly modified form by the Byzantines. This notation used letters of the alphabet to stand for 1s, 10s, and 100s: the letters α-ι stood for 1-9, κ-ϙ stood for 10-90, and ρ-ϡ stood for 100-900; after that, ,α-,ι represented 1000 to 9000.
(ϙ and ϡ were archaic characters that dropped out of use for every other purpose; so was ϛ, the symbol for 6. These letters were retained solely to keep the number of available numerals up to 27.)
So for example
For numbers above 9999 you might expect them to extend the tick system (e.g. ,ι = 10,000), but instead they borrowed from the second system (see below) and used Μ to refer to 10,000, with multipliers above the Μ if needed. This is simply because there was little practical use for numbers beyond that point. Sometimes ι-ϙ and ρ-ϡ could be used with two dots above the numeral to represent 100,000-900,000 and 1,000,000-9,000,000, but that wasn't a universal standard.
The second, and older, system available was the Attic, "acrophonic", or "Herodianic" system. This was similar in design to the Roman system. Ι = 1, Δ = 10, Η = 100, Χ = 1000, and Μ = 10,000 (this is where the Μ in the Milesian system came from). Numerals for 5, 50, 500, etc would be constructed by writing a Ι, Δ, Η, Χ, or Μ inside the legs of a Π. Most of these symbols come from the first letter of the word for each number (deka = 10, hekaton = 100, chilioi = 1000, myrias = 10,000, and pente = 5). So for example
I can't write the numerals for 5, 50, 500, etc. because the symbols are not supported by common computer fonts; you can see some examples here, though. This system was widely used up to the 300s BCE, but was gradually replaced by the alphabetic system; it remained standard in official Athenian inscriptions until 100 BCE or so.
There is one other minor system to note: a system of "octads" invented by Archimedes specifically for representing extremely large numbers. This one was obviously never widely used. In this system the first "order" represented numbers in the range 1 to 10^(8) (i.e. a myriad myriads); the second order represented numbers in the range 2 x 10^8 to 10^(16); and so on up to the eighth order, which went up to 10^(64). That constituted the first "period", which amounts to working in base 10^(8). After that would come the second "period", but I can't reliably work out how to express these numbers in decimal notation after this point. The New Pauly states that it could represent numbers up to "a 1 followed by 80,000 billion zeros", but I have a nasty feeling that may be the older European "billion" that was 10^(12) rather than 10^(9).
Finally, fractions. These were poorly notated. Reciprocals, that is to say fractions of the form 1/x, could be expressed by writing x as a numeral followed by a tick at the top right, e.g. γ' = 1/3. A sign that looks a bit like L' was used for 1/2, and β' = 2/3. Fractions with a numerator other than 1 could be written by putting the word or numeral for the numerator before the denominator: so e.g. 3/11 could be expressed by τρία κα' or γ κα'. Diophantos adopted the more visually effective system of putting the denominator over the numerator (opposite to modern practice), so he would express 3/11 by writing κα' over γ. Ptolemy's cartographic work the Geography used a form of this system of fractions in base 60, representing minutes and seconds of arc with a single and double tick respectively, as we still do today. The base 60 thing was of course indirectly inspired by Babylonian mathematics. He may have got this system from Eratosthenes, or alternatively Hipparchos. His notation (and therefore, perhaps, Eratosthenes/Hipparchos) is notable for including a notation for zero, Ο, but since it was never used outside cartography/astronomy, that didn't catch on in popular use.