What did the elites and urban masses do for entertainment? Were there any competitive games during the festivals?
So, I'm not an actual historian but I'll try my best to follow the guidelines in answering your question.
First, yes, Imperial China had many physical sports for both leisure and practical purposes
Firstly there is the sport of cuju/Ts'u-chü which is a game that involves kicking a ball through an opening in net to score points that also forbids the usage of the hands. If this sounds familiar, that's because it very much sounds like a basic description of the game of Football, something that even FIFA has noted. That said, the actual rules of the game were a bit different to say the least. In the variant with a net, according to a Song dynasty writer name Wang Yuncheng, two teams would take turns with the ball, with one team passing the ball between themselves before ending with the team captain kicking the ball into the net at which point the other team would get the ball. There were even cuju clubs where the sons of wealthy families could go and play, as well as professional players that could be hired to beef up a team.
There is also evidence that the Chinese also played a game similar to golf called chuiwan. Whether or not this game is actually the ancestor of the modern game of golf I won't get into, but suffice it to say the game has more than a passing resemblance to the Scottish sport. The game saw it's peak during the Song dynasty, continuing through the Yuan and Ming before declining during the Qing. Both chuiwan and cuju eventually also made their way beyond the imperial court and into common life, and cuju is still played to an extent today with the game jianzi.
A game that was popular during the Tang dynasty, before China lost access to good horse pastures in the north, was Polo. No wonder considering the Tang's nomadic descent and access to Central Asia via expansion and trade routes. Specifically a past time for the wealthy, there is evidence of both traditional horse polo as well as donkey polo as just last year a noblewoman's tomb was found with multiple donkeys and riding gear buried alongside her. Men were specifically encouraged to play to improve their horsemanship in case they ever need to go into battle, but the game was also very popular with the women in court.
More generally, both dancing and gymnastics were common entertainment, though only the former was generally considered proper for a women of the court. In dancing, there were both traditional dances alongside sword dances, the most famous example being Xiang Yu's attempted assassination of the King of Han Liu Bang aka Emperor Gaozu of Han disguised as a sword dance. Sword dancing was also very popular in the Tang dynasty, with the poet Li Bai waxing on the subject in multiple poems.
On the more martial side, China has a long history of martial arts that were practiced and showcased in matches for the entertainment of the imperial court. Shuaijiao, a form of jacket wrestling, is the modern incarnation of a lineage of wrestling that traces it's routes back to at least the Zhou Dynasty. That, along with proximity to Mongolia and other steppe cultures meant that northern China, especially during the Manchu ruled Qing, was a hotbed for wrestling. The Kangxi Emperor even established a wrestling school which trained guards but also to wrestle for the entertainment of the Emperor. It is from this school that modern shuaijiao descends.
As well, there is also Wushu which traces the origins of it's many styles from military training to other sources like the Shaolin Monastery or Taoist Daoyin exercises. Though only codified in the Ming and Qing dynasties, just like in HEMA, there are centuries of martial arts that were practiced that we know almost nothing about. There is also archery, which, along with horseriding, were among the subjects tested fro during imperial and military examinations with a tradition that also traces it's origins to the Zhou Dynasty.
There is even a tradition of weight lifting in Chinese history, with the military specific examinations instituted by Wu Zetian having a session where the test taker had to lift a stone weighing from 120-180 kilograms 35 centimeters from the ground.