When Henry VIII granted a title to Anne Boleyn, why was she styled as "Marquess of Pembroke" instead of "Marchioness of Pembroke"? Have there been any other occasions where a monarch bestowed a typically 'masculine' title to a woman?

Actually, the Marquessate of Pembroke was the first ever hereditary peerage ever given to a woman in British history, so it's significant for that reason, as well. It was given to Anne, by Henry VIII by the masculine title likely because the Marchioness title is reserved for the wife of a Marquess, and there was no Marquess of Pembroke theretofore in existence (it had previously been an Earldom) at all. When Henry created the title, he chose Pembroke as it had strong connections to his family -- the prior Earl of Pembroke had been his great uncle -- and he gave it to Anne clearly because he intended to marry her and he wanted her to have sufficient nobility to be worthy of marrying a king.

It's worth noting that the title contained hereditary succession through male heirs, so any of Anne's sons (legitimate or otherwise) would have inherited the Marquessate, the same as if she had been male. This is interesting, because it treats Anne almost as if she had been a Marquess herself, despite the fact that she was a woman. The Marquessate of Pembroke ceased to exist with Anne's execution, of course, so we don't know how succession would have continued, but it is clear that this is a unique and interesting -- and groundbreaking -- case.

As for other occasions in which monarchs have given male titles to women, I am not aware of them, simply because the terms are usually kept along gender lines -- this one does happen to be pretty unique -- but that may be more of a linguistic factor than a historical one.

Source: Warnicke, Retha (1991). The rise and fall of Anne Boleyn: family politics at the court of Henry VIII. Cambridge University Press.