They arose out of the names of the amateur athletic clubs that begat the first professional baseball teams.
Here is an article from the New York Times on December 19, 1854, reporting on the three amateur baseball clubs in the city. By name, they are the Gotham Base-Ball Club, the Knickerbocker Base-Ball Club, and the Eagle Base-Ball Club. The players referred to themselves respectively as the Gothamites, the Knickerbockers, and the Eagles.
By 1860, metro New York boasted such powerhouse amateur clubs as the Excelsior Club and Mutual Club in Manhattan, and the Atlantic Club and Eckford Club in Brooklyn. There were more than fifty such clubs in the area, and since most competition was local, referring to a club as "Brooklyn" or "New York" would have been confusing.
Instead, newspaper accounts would call them "the Atlantic club", "the Atlantic", "the Atlantics", and the "Atlantic's" (as in "the Atlantic [club]'s [players]) pretty interchangeably, as exemplified by these two (1, 2) 1860 news articles published a month apart.
It took a little bit longer for this practice to catch on outside of New York. Here's one example of an AP article from November 30, 1866, which didn't yet follow the practice.
The practice still was not fully adopted by the time of the formation of the National Association in 1870, the first pro baseball league, as evidenced by how its teams were referred to in the newspapers.
Six years later, when the National Association folded and the National League formed (which still exists today) all its charter members were using the team nickname convention.
Having a cuddly mascot came later.
This convention was not followed at the same pace in Europe, which is why older football clubs have names like "Westhavershamshire F.C." It's mostly the newer clubs in Europe that have nicknames and mascots, having adopted the American convention.
EDIT: And when I say these were "amateur athletic clubs" in the beginning, they literally were exactly that. Anybody off the street could pay to join, just like you could with an adult sports league today. The club would play games against itself as well as schedule games against other area clubs. If you browse through newspapers of the 1840s, 50s, and 60s, you'll sometimes even see advertisements inviting members of the public to join these baseball clubs. So the names of the teams were originally just the names of the clubs, and the names of the clubs were pretty arbitrary. For example, the Eckford Club of Brooklyn was named in honor of a local hero of the War of 1812, while the Atlantic Club of Brooklyn was probably named that because their home field was near Atlantic Avenue.