If you had private transportation, you were probably pretty well-off. You also probably wouldn't own a single horse for riding. Carriages were more common than mounted riding, and very few people were able to own one. If you were so lucky, you probably kept the coach at a boarding stable, where other people could take care of the horses (which required a lot of maintenance and produced a great deal of waste).

If you were looking for public transportation, you lived in a pretty exciting time. There would probably still be a lot of omnibuses. These were big horse-drawn coaches that fit fifteen people and operated like a bus. There might still be a few horsecars around, which were larger coaches pulled by horses along iron tracks. However, those would be giving way to a brand new form of transportation: the cable car.

Steam-powered cable driving plants would keep a cable running around a fixed route. Cars could grab the cable when they needed to move forward, and release it when they needed to stop. These didn't require horses, and could go up steeper inclines than the old coaches. This meant much less upkeep and expense, and less waste on the streets.

If you wanted to travel longer distances, there were some elevated steam railways.

Over the next decade or so, electric trolleys would replace some cable cars, subways would be experimented with and quickly become a reality, and elevated lines would expand. You would be standing right in the middle of a major period of growth for urban transportation.

More information from the New York Transit Museum: http://www.transitmuseumeducation.org/trc/background

Sources:

Jackson, Kenneth. "Technology and the City: Transportation and Social Form in New York." Annals of the New York Academy of Sciences 424, no. 1, 283-87.

Morris, Eric. "From Horse Power to Horsepower." Access 30:2-9. University of California.