I know that baccarat was popular among the British upper class at the time Fleming wrote the novels, but why and what does Bond playing the game tell the audience?
Two Ian Fleming questions in one day. Weird.
So you actually kind of answered those questions yourself. Baccarat was extremely popular in Britain in the 50s and 60s and Fleming wanted to make sure Bond's card game of choice fit the tastes of the time. There are a couple other reasons why Fleming would've chosen baccarat specifically. For reference, I'll be drawing on Fleming's own work, gaming trends in Britain at the time, as well as Andrew Lycett's biography of Fleming.
First, familiarity. We already covered that baccarat was popular in Britain in the 50s and 60s. However, Fleming himself had a familiarity with the game. Lycett reminds us of a trip that Fleming took to the Estoril Casino in Portugal while en route to the US during Fleming's time in the Naval Intelligence Division in WWII. He and the Director of the NID, Admiral Godfrey, played several games of the chemin de fer variety of baccarat. It's likely Fleming included baccarat in his work not only because it would've been widely recognized and understood, but also because he had a working knowledge of the game. Further supporting this viewpoint is the fact that in the 2006 movie version of Casino Royale, Bond plays Texas Hold 'Em instead.
Second, the nature of baccarat. Baccarat is a complex game, but it doesn't require much skill, the banker isn't permanent, and hands can be played quite quickly. The fast play and lack of skill lend themselves to baccarat being talked about in media. In Casino Royale, when Bond walks up to the table La Chiffre is playing at, Fleming devotes a lot of the final scene of that chapter to just how baccarat is played,
"‘It’s much the same as any other gambling game. The odds against the banker and the player are more or less even. Only a run against either can be decisive and ‘break the bank’, or break the players'."
That's a small excerpt, but Fleming actually goes on for several paragraphs with La Chiffre explaining how to play baccarat to Bond and Bond asking questions. However, when they're actually at the table playing the language changes,
"From the decision to stand on his two cards and not ask for another, it was clear that the Greek had a five, or a six, or a seven. To be certain of winning, the banker had to reveal an eight or a nine. If the banker failed to show either figure, he also had the right to take another card which might or might not improve his count. Le Chiffre’s hands were clasped in front of him, his two cards three or four inches away. With his right hand he picked up the two cards and turned them face upwards on the table with a faint snap. They were a four and a five, an undefeatable natural nine. He had won."
What was honestly a very laborious scene in which La Chiffre pretentiously "teaches" Bond how to play Baccarat gives way to a very fluid scene describing Bond's analysis of the game and the player's strategy, as well as La Chiffre's winning hand. This scene would be much more difficult to describe if the characters were playing something more akin to even Poker.
Continuing on this point, Fleming could've very easily chosen Blackjack as the game of choice. It's incredibly easy, low skill, and easy to describe. However, it's the social nature of baccarat that makes that game in Casino Royale so compelling. If you're playing Blackjack, and you're at 18, you're gonna think really really hard, and likely decide not to hit it for fear of going bust. But that decision on whether or not to hit it or not is not influenced by the other players at the table. In baccarat, however, your decision on whether or not to accept or refuse a card can be greatly influenced by the other players, whether they choose to accept or refuse a card, and who the banker is. Because the banker changes throughout the game and because you're all trying to beat the banker for the same bank (pot), this kind of reinforces the basic mathematical rules of Blackjack (the higher a number you have, the less likely it is that the card you need will come up) through a kind of social sanctioning. Everyone else is only accepting cards when their hands total 0-4 and everyone is only refusing cards when their hands total 6 or 7, therefore, you're gonna be way more likely to do the same and you're gonna know what a player's hand is roughly by whether they accept or refuse a card. All of a sudden this game of relatively little skill seems much more dynamic and nuanced.
"The Greek pushed forward five plaques of one hundred thousand and the croupier added these to Le Chiffre’s half-million plaque which lay in the centre of the table. From each bet the Casino takes a tiny percentage, the _cagnotte_, but it is usual at a big game for the banker to subscribe this himself either in a prearranged lump or by contributions at the end of each hand, so that the amount of the bank’s stake can always be a round figure. Le Chiffre had chosen the second course. The croupier slipped some counters through the slot in the table which receives the _cagnotte_ and announced quietly: ‘_Un banco d’un million_.’ ‘_Suivi_,’ murmured the Greek, meaning that he exercised his right to follow up his lost bet.
Bond lit a cigarette and settled himself in his chair. The long game was launched and the sequence of these gestures and the reiteration of this subdued litany would continue until the end came and the players dispersed. Then the enigmatic cards would be burnt or defaced, a shroud would be draped over the table and the grass-green baize battlefield would soak up the blood of its victims and refresh itself. The Greek, after taking a third card, could achieve no better than a four to the bank’s seven. ‘_Un banco de deux millions_,’ said the croupier. The players on Bond’s left remained silent. ‘_Banco_,’ said Bond."
Fleming's description of each hand is incredibly quick. Often only using 1 or 2 sentences. While the rest of the dialogue is taken up by the drama of the players, mimicking how a baccarat game might feel like in real life to the reader.
Fleming sets the stage for the drama of this scene by first telling you the rules of baccarat ad nauseum in a literary device meant to display that La Chiffre does not know that Bond can play baccarat very well, then by having Bond sit back and regard the other players of this game, before finally joining himself.
In any other casino card game besides baccarat, the drama of this scene would be lost. Blackjack doesn't make the players engage with each other enough for drama to happen and in Poker, the drama is silent and hidden behind the player's bluff.
Third, sophistication. This one is a little more ephemeral, but indulge me. Bond is literally meant to be the height of sophistication. He is based heavily on not only Fleming himself (trying to tip his own hat), but also on heroes in pulp novels like Bulldog Drummond who exemplified what people would have regarded as the epitome of upper class sophistication and manner for the time period. Baccarat and the chemin de fer version specifically was imported to England from France. Primarily wealthy British officers stationed abroad in WWII picked up a fondness for it and brought it back to England. Keep in mind, Fleming's principal experience with the game came from his time abroad in the NID during WWII. The game gains popularity in Britain and starts to take on this air of sophistication due to it's association with the upper class, France, and English seaside resorts. All of this combined makes it seem natural that Fleming would want to associate his suave and sophisticated protagonist with baccarat.
So why baccarat and what effect would it have had on Fleming's audience? In short, the game was popular and Fleming had a working familiarity with it. The simplicity and drama of baccarat lend itself to literature. And baccarat had an air of sophistication which Fleming would've wanted readers to associate with James Bond.