I think I started to read that book long ago, and didn't get very far. How is this insinuation presented? What exactly was it talking about?

European music has had some things that make it quite different from other forms of music. Some of those things are indeed very related to "massed" or "mixed" instruments/voices. However, it doesn't mean other musical traditions don't combine instruments at all.

The guqin is an example of an instrument that is (as far as I know) played by itself. It was the instrument of the learned men in China. The biwa was also played without other instruments in Japan. However, we find ensembles in Chinese and Japanese musical theatre, so we can't really say they didn't have music with mixed instruments.

Gamelan music has been made with several instruments in an ensemble.

Asian music is way above my pay grade. I only mention those cases as an example (and I might not be 100% right about them). You might want to try /r/ethnomusicology, perhaps /r/WorldMusic, for more information.

Now to the peculiarities of Western music.

For quite some time before 1500, European music had this peculiar thing we call counterpoint. As far as I know, other musical traditions did use several instruments and/or voices at the same time, but did not get into what we call counterpoint. What is it, exactly? Well, you take two or more individual voices/instruments and have them play different parts/melodies at the same time. One note (punctus) is played against another (punctus contra punctum). The possible combinations (both in terms of duration and pitch of the notes) were studied and eventually codified (as in creating rules of how you are supposed to properly make one melody work with another). You didn't just put whatever you wanted together and call it counterpoint, there were proper ways to do it. This was the way the learned musicians were composing "serious" music.

The first 5 minutes of this video present a melody that is ornamented, sung at times by only one voice and then by several voices, but everybody is singing the same thing. Other musical traditions have that kind of participation. Now, listen to what happens five minutes later (and please turn UP the volume before clicking on that link, to really get the punch this music packs). We can hear four distinct parts (each one can be sung by more than one person) with different rhythms, directions, ornamentations. That is counterpoint from 14th century France, the kind of "mixedness" that came to mind when I read your question.

Not everything was contrapuntal or actually written down in European music. There was some music that required improvisation or some form of more or less fixed accompaniment for a sung melody (here's some music from the same French composer).

As far as "massed" mixed music, what we consider a large ensemble now dates to, at the earliest, Handel's massive productions such as the Messiah, shrinks down again with Haydn and Mozart, and swells back up with beethoven, starting at the earliest in the mid 1700s with Handel, whom beethoven admired greatly. And as far as we know from primary sources and anecdotal accounts contemporary to the period, the first conductor who was picky about tuning was jean-baptiste Lully, and he also was responsible for the expansion of acceptable ensembles to larger sizes and allegedly introduced double reeds into mixed consort as called for by the composer. During the Renaissance, oftentimes instrumentation was unspecified as pieces were presumably played on whatever was handy, with mixed voices, viols, and plucked strings, or combinations involving organ and brass.

But what we call orchestral music in modern practice, as stated above, really dates to the very late baroque atnthe earliest, placing it around 1760. Music of the 1500s is usually referred to as chamber music. Did the book itself use the terminology "orchestral"? That would be problematic for a number of reasons.