I think there are a few misconceptions within your question. Small ensembles, often consisting of 4-6 musicians, have been around for hundreds of years, and were especially popular in the Baroque era of classical music (Bach, Vivaldi). During this time period, music was generally commissioned by nobles for social events; it would be impractical to hire an entire orchestra for a dinner party.

Large orchestras gained popularity in the classical and romantic eras (Mozart, Beethoven). With the rise of operas, people began going out to hear music, rather than hiring a few musicians, and classical music gained popularity with the lower classes. Many composers used this opportunity to create pieces for large orchestras. However, ensembles remained popular, both for composers and audiences.

The idea of the "Band" starts in the early 20th century. Jazz, which was just gaining popularity, combined the ideas of European classical music with traditional African music. European music was generally composed, then performed, often without the original composer. In jazz, however, the piece was often either composed and performed by the same people, or improvised on the spot. Prohibition in the US helped, with many speakeasies hiring a small number of jazz performers, as did the great depression, which made large bands too expensive to be practical. All this encouraged smaller, tighter groups.

There are certainly exceptions, most notably big band music. The 1942–44 musicians' strike increased the emphasis on singers, rather than the instrumentalists as before, and helped push modern music away from large groups. Effectively all modern popular music has its roots in jazz, and inherited the concept of smaller bands. Keep in mind though that this is only the trend, not the rule; folk music pre-20th century would often have only a few performers, and modern classical composers still use large orchestras.

As a final note – “band” and “orchestra” are fairly nebulous terms. I've always understood band to mean “without strings” rather than “smaller than an orchestra”, although the term is used in a large variety of contexts.

This is brief and generalizes a lot, so I can get into detail if you want. I'm no expert on music, but I have done some research, and can provide sources if necessary.

You've set yourself up kinda a false dichotomy here! Music ensembles of small sizes in Western music have always been there, they just didn't have a band name or really an identity as a "band" past more than the performance or maybe a few performances. I think a lot of the false impression of having big ole orchestras 24-7 comes from the dominance of large orchestral music in popular CDs, NPR stations, and what sort of performances they take kids to on field trips. Unless you're frequenting like college-level music recitals and the dark depths of small-views Youtube videos, you're not likely to get exposed to the small stuff! Here's a very popular Classical example of just five instruments.

There's also a common thing of "souping up" music to big orchestral sizes, take for instance that Handel's Messiah these days is popularly performed "American Style" with a big brassy orchestra and a giant Mormon Tabernacle-sized choir. Here's a Historically Informed version which sounds downright pitiful next to this balls to the wall version with like a billion more musicians. Don't get me wrong, I love listening to a big brass Messiah, but that's not really how it was done when Handel was in charge. And if you've got the money why not add a lot more musicians? Handel would have!

The bigger thing in your question is when musicians began to be part of bands or orchestras as their identity instead of free-agents that took gigs, to which you might find these older answers interesting! Check out /u/origamitiger's answer in particular.

You need to be careful in your assumptions. The 18th century brings the rise of the large orchestra, but chamber music never went away. Beethoven and Handel and countless others arranged songs to be played in the home as a way to supplement their incomes.

Also during the Great Depression in the US following the Stock Market Crash of 1929, bands were downscaled, so there was a shift from 20 piece big bands where each member required adequate payment to smaller 4 or 5 piece combos where the group could survive being payed less as a whole. We can see that in the 20's big bands were really popular but after that they sorta just fade away from the public spotlight. In regards to the singular singers though I'm not sure, but maybe it might be that singers would get popular by singing with a band and they would be the person that the public would most easily associate with since you hear their voice during their performance just making it more personal. And then as they rose in popularity, the economic side of the industry realized that marketing it as a group wouldnt be as effective as marketing the main, well known singer. Just a thought.

There have certainly been a number of shifts in the types of groups that perform music, but I think it is important to note that music played by a small group, as opposed to a soloist or an orchestra, has existed and been popular for a long time. Chamber music, for example, has been played since the Baroque period, with a small number of instruments playing different parts, and could be played conveniently at social gatherings.

The current "rock band" is kind of descended from the "jazz combo" which is kind of descended from Dixieland bands which are kind of descended from military/brass bands. I'm sure it's more complicated than that, and "real" historians could fill in a lot of the details, but check out a video series by Wynton Marsalis called Marsalis on Music, specifically the segment "Sousa to Satchmo".

I'm going to pop in as an amateur musicologist and try to clarify some problems all around.

As has been mentioned, there's a false dichotomy created by the question; music has never been divided, historically, between small and large groups. It wasn't the case that in the 1500s there were only fifty-piece orchestras, and currently there are only 4 piece groups. It's more a matter of function, and composition. While there were few large groups before the Renaissance, they did exist on occasion.

On function: the purpose of the musical group was important. If you need some music for an important function, where people will generally be talking, or dancing, the group doesn't need to be particularly large, and you can save a lot of money by having a smaller group. That being said, if you're particularly rich, you may have wanted to show off with a large group. Similarly, you may have had enough money to employ your own orchestra (I'd love it if the ultra-rich would do this in modern times...), and composer. Lully had a larger orchestra (for the time) to compose for. Similarly, big bands in the jazz era could afford to pay their musicians, by getting decent pay for the gigs they did.

So, it depends really on the function. You're also grouping the entirety of music somewhat poorly; just because society as a whole remembers classical music as orchestral, doesn't mean it always was; as mentioned, there were smaller groups throughout history, from the soloists (organists in churches, like Bach, to quartets in Nazi concentration camps, like Messiaen's). But not just this, there were different types of music going on. While Beethoven's 9th symphony was being performed, an older woman in a Newfoundland outport was singing She's Like the Swallow. So there are the questions of "popular where" and "among whom"?

There was also a good deal of popularity of the individual, rather than the music, as well (or rather, an individual became popular by composing good music, sort of, sometimes...). So yes, Beethoven was popular in his time, and he wrote for symphonies as well as string quartets, concertos and piano sonatas.

The idea of the band has been around a lot longer than we can really know. The term "band", I don't know in terms of etymology, but the idea, well, that's one of those things much more difficult, historically, to pin down; in the pubs and drinking halls, there would have been music to dance to, performed by a small group (depending on the pub or drinking hall, of course). This would fit your idea of a "band", though they would have been acoustic through most of history, and wouldn't have become terribly popular (though they might have performed songs well-known to those who lived in the area).

So, your question is problematic, as it assumes divisions that really didn't exist; we may only talk a great deal about the great composers of orchestral music, but a lot more music was happening throughout history, and ethnomusicologists have worked to preserve some of that, or research more of what was happening at the time.

Thought I would edit to add that you may want to read more on the history of western music, Grout and Palisca's text is considered one of the standards.