Perhaps the most interesting examples of non-circular coins are those in semi-Hellenised India. Take, for example, this mid-2nd century BC piece from Bactria: a square drachma! It's a perfect encapsulation of what made the Greco-Bactrian state so unique – Indian imagery and coin shape, combined with a Greek legend and weight-standard.
Going further eastwards, Chinese copper and bronze coins have traditionally included a square hole, and eschewed complex imagery in favour of small legends. Take this coin of the Taiping Heavenly Kingdom, with the obverse (left) displaying the characters for Taiping Heavenly Kingdom, and those on the reverse (right) 'Heavenly specie'.
Even in modern times, coins have not always been circular. One need look no further than the UK, which introduced the septagonal 20-pence piece in 1982, and (thankfully just outside the 20-year rule) the also septagonal 50-pence piece.
Why exactly were coins round? Well, I'm afraid I can only disappoint you in this regard, as I can only really explain why they don't have to be. Ancient coins were generally minted using two dies (mould-like sections that are in effect negatives of the actual image) – one a moveable piece for striking the obverse, which would be a tall, semi-cylindrical piece which would be struck with a mallet at the top, the other embedded in an anvil. (See this diagram.) This meant that theoretically, you could (as the Bactrians did) mint square coins if you had square dies and square blanks. Meanwhile, Chinese coins were not minted but cast, leaving behind a sprue to be trimmed off, which possibly accounts for the lack of fine detail but more importantly enables the existence of the characteristic square holes. I'm afraid you'll have to look to a better numismatist (or indeed metallurgist) than I for an explanation of how solid round coins came to dominate.
Sources:
