There is a name and entire practice involved in the study of flags, its called vexillology (latin root). Its way, way too complicated to reduce into a single post, so I'll narrow my answer to just describing a flags function and form in known history, as a general jumping off point so YOU can better narrow your research and find answers, because I cannot by any means answer this to any one end. I majored first in art history, but I'm not an anthropologist. You've asked for several books worth of knowledge but that's not an insult when knowledge is the goal!
I'm answering his because I bet this can seem like a hard topic to learn about. I tried looking up some notes I had taken about this topic in class because when I Googled this topic, all I got were a bunch of kids links on he American flag. Without knowing the word for "flag history," you could get lost easily in padding. And the books can be scarce.
No one can say what flag was the first ever. The Netherlands say it has the oldest tri color flag, but its design came from Charlamagne in the 9th century. The Ottomans were using the Crescent on their banners before that. Trade flags have designs that go back even farther. Its a lot easier to track the design of a specific flag than the idea, because until recently, it was almost impossible to track the specific history of an idea. Thats why tracking the art and history of a certain flag or symbol is easier and more comprehensive, I feel as if I'm painting with too broad of strokes but I can't explain the history of flags without explaining about seventy other ideas and concepts of communication through symbols.
So: super narrow reductions of flaghood to follow. I'm only going to explain where the concept of flags came from generally, and when and how they became the symbols for countries as we know them now. I'll also not go too deep into art history.
A flag is some of the earliest human work in branding. A flag, like a banner, family coat of arms, or a shield, is more than just a way to let everyone know who's side anyone is on in battle. Its evolved as a way to signify the statehood, art, and national identity of a country. It's important to understand how flags became national symbols during the rise of 18th century Western nationalism because flags were not always a specific signifier of statehood. They're older than our modern ideas of what they mean.
Flags are not likely a Western invention, even if they became a Western symbol of nationalism. The honor belongs to the Arabs and the Persians, although the idea of flags spread West and far East, and even into Africa. Flags were used to signify armies in battle and used to signify guilds in much the same way banners and sigils were used. They were made of cloth, metal, and were usually designed to be simple and strong. That is the worst reduction of flag history and art ever, but just understand: flags are old branding.
Flags as we know them artistically, the rectangular symbol and held high, have been used since at least the third century BCE in both Arabic and Eastern culture to mark allegiance in battle. But in general, and the way we see it *now,*until the 1600s and their adoption into seafaring and empire (specifically by Great Britain), flags were more of a symbol and a banner than any specific call to nationalism. Every country didn't always have a flag. States and nations had ideas of flags, and even specific flags, as well as banners and arms, but seafaring is when flags as we understand them, for symbols of nations and empires, really got hot and became the international must-have. That terrible reduction is meant to communicate this: flags got tied to international travel and communication.
Flags adoption into signifiers of statehood is recent and it has a lot to do with the adoption of maritime laws and shipping routes in the 1600s. Flags were a way to signify what port of origin and greater nationality or company a ship was part of, so they didn't get attacked at sea. This sometimes made ships easier targets. However, pirates faced huge penalties for flying under a "false flag" or attacking a ship protected by a wealthy nation or company. Stealing a ship was especially risky because countries tended to produce distinct ships and governments kept very good maritime records of vessels. But in the age of sail, as empires expanded and companies plus countries expanded and defended their shipping routes, flags started to become a must have for any on the go country of empire! And nationalism played a big part of that in Great Britain. Reduction takeaway here is: Flags got a useful function and brand message we know even today.
Flags in state got hot in he 1700s as the world saw the rise of nationalism get turned to branding. The printing press may have helped this along in a popular sense, but thats far from a sure thing. Britain helped catapult the flag of a nation to standard with the popularity of the Union Jack, but they were far from the first country to adopt any one flag in their history. Great Britain was just the first to do it for the communication purpose as we know and apply it today, and even then, they were just revisiting an old idea.
I hope that jumping off point helps. Flag history is fascinating, but its so complicated, its impossible to answer this question without cutting out huge swaths of history and culture. But i hoped I helped you narrow your scope at least, and aided your further research. Vexillology is a hell of a scope.
Good sources:
The Red Flag A book about the origins of communism. It actually goes into flags and statehood quite a bit though, and it might help you understand why certain symbols mean and communicate so much.
The Corporation That Changed the World
Whitney Smith is a great author and has written many books on the subject as well
addendum question, what is the first known flag?