It does not seem like there was a formal rule against crossdressing in the Gran Colombia of the age. It seems, rather, that crossdressing was frowned upon as part of general societal rules that maintain gender roles. The high society of Gran Colombia, which naturally would include El Libertador, was trying to consciously imitate the fashions and styles of Europe. For example they tried to shake hands "a la inglesa" or cheer Bolivar with cries of "hip hip huzza". Though there were properly Colombian music and dances, balls celebrated in Gran Colombia would still follow a European style, and the people were expected to conduct themselves in a certain way. It would certainly be improper for a lady to wear men's clothes, and for a man to wear woman's clothes, because doing so would go against the established gender roles that members of high society must follow.

These rules would not be contained to balls and celebrations, but any respectable person would be expected to follow them in his day to day life. Bolivar, as a wealthy criollo educated in Europe, naturally keenly felt the need to behave respectably. By and large, it seems gender norms and societal expectations were similar to those in Europe during the early Republican period. Thus, ladies were supposed to be polite, defer to their male relatives, and keep to the private sphere of home, whereas men were more assertive if still polite and had to be in charge of the public sphere of politics and government. Consequently, a good and proper lady should have appeared at the ball dressed appropriately and in a calm, level-headed and pleasant mood.

Now, Manuela Saenz was not a proper lady in any sense of the word. Ah, Saenz... now that's a woman. Despite the fact that Saenz was also a privileged criolla brought up in high society, Saenz completely rejected the expectations that the age had for women. She seemed to delight in scandalizing polite society, going out of her way to defy societal norms and attack the old system. For example, she was married to an English gentleman yet made no effort to conceal her affair with Bolivar. She would spend time with him in Lima, while her husband was but a few houses away. El Libertador certainly had the right to have several mistresses, no one would reproach a man for that, but for a lady to cheat on her husband? Bring me my smeling salts!

This was not everything. She also enjoyed to leave her dresses aside and instead dress in a military uniform. Yet another action that scandalized polite people, for a lady was not supposed to dress like a man and intrude into the sphere of politics and the military. She would go around town dressed in pants (the horror!) and carrying pistols or swords, and instead of being demure and polite she would try to intrude and talk politics or reproach people for not showing enough deference to Bolivar. At one time, she personally went to stifle a military revolt, and in another, more famous assassination attempt against Bolivar (which took place after the masquerade ball you describe), she tried to face the conspirators herself - and was pistol whipped for her troubles.

There are several even more bizarre stories involving Saenz and her desire to defy all societal norms, including her keeping a pet bear or taking a corpse's mustache and wearing it. The important thing to know is that Saenz consciously and continuously tried to go against the societal norms of the age, which include dressing as a man. Proceeding to the ball itself I consulted some sources and they claim that Saenz was turned away for how she dressed, so she had to return home and put on a dress. But she dressed as badly as she could. There are no details except that she was "desaliñada", but one can infer that she did not do her hair, didn't use an adequate dress and did not respect the myriad other norms that one should observe when attending a formal ball.

The sources (all in Spanish), then claim that Saenz was allowed entry once she dressed like a woman, which makes it seem like dressing as a man was a greater offense. Being that dressing as a man was not merely a faux pas but a statement that she was equal to men and could be part of "their" sphere, then yes, it is a greater offense. Remember, Saenz would not merely appear in any male suit, but in a military uniform. A woman dressing as a man was bad enough, but to dress as a military officer? Completely unacceptable. In any case, once she got inside Saenz "caused a scandal". Again, no details are offered but she probably started to scream and argue with Bolivar to try and convince him to leave the masquerade - yet more inadequate behavior. That finally embarrassed Bolivar enough to abandon the ball.

So, there was a rule against breaching the expected gender roles, which was part of a wider series of norms that sought to dictate the proper behavior of women and men in polite society. For Saenz and the people around her, dressing as a man was more than merely impolite. It was a statement, a way to openly defy society and its norms, and thus unacceptable. People did not commonly dress up as the opposite gender because that would go against most societal norms. Saenz dressed as man because she wanted to defy those norms, and the ball was not the only occasion she did so, often parading around cities in an official's uniform. Knowing Saenz, if dressing as a man had been acceptable in any way, she wouldn't have done that as often because her goal was scandalizing people and breaking expectations.