The top hat most likely became a magician's prop as a result of it being a popular fashion accessory of the time. Magicians wore tuxedos when that was popular evening wear. They weren't setting out to appear different than the audience, but rather to wear popular clothing. The top-hat would have been part of that outfit and was most likely used because of that.

There is nothing inherent about a top hat that lends itself to magic, besides perhaps being large and rigid enough to hold items without those items changing the outside appearance of the hat.