The "purpose" of what McQueen does there is to up-stage Yule Brynner--that whole scene, actually, is really just the two of then competing for the limelight (which maybe makes it more impressive, it adds to the tension in the scene). McQueen was notoriously competitive, especially in that film in particular--numerous interviews from the actors have pointed out how McQueen would obsess that Brynner was getting more attention. Shaking the shells was an ad-lib (according to Eli Wallach's autobiography), and provided it's something that McQueen didn't just feel like doing to divert attention from Brynner it may be something McQueen picked up from when he was a Marine. I've heard of shooters shaking their shells to loosen the charge and ensure a cleaner shot, but I'm pretty sure that's an urban legend since shotgun shells work just fine either way--in any case I don't even know whether the gun McQueen uses would even have hard shells, or if it would've been using paper shells