Most people believe that she and her brother were NOT lovers, that they were set up by people wanting to bring the powerful Boleyn family down. After waiting for years to become queen and finally achieving it when Henry married her in 1533, there is no way she would risk becoming pregnant by anyone other than the king--what if the child should resemble the man she had an affair with?
Anne and her brother George were very close; Mary, not so much. Her sister Mary was Henry's mistress for a short time (and may have had at least one child by him, but that is for another thread), then was married off to Henry Carey. Anne was presented to Henry, who quickly became enamored of her and wanted her for his mistress; she wanted to be his queen and told him so. After the break with the Catholic Church and his divorce from Catherine of Aragon, he married Anne, who gave him a daughter (Elizabeth I). Three other pregnancies followed, all ending in miscarriages, including the son Henry desperately wanted.
By this time, Jane Seymour was being dangled in front of Henry. She was everything Anne was not: demure, obedient, from a good family, etc. So now something was needed to dismiss Anne. Henry had risked everything to get Anne, so she couldn't be tossed aside with an "I'm tired of her now" attitude. Plus, Henry was now the head of the Church of England. Something that went against morals and church behaviour would be good to charge her with. What better than saying the queen had had numerous affairs, with incest to boot? Thomas Cromwell was tasked with coming up with something to charge Anne with, and adultery was the ticket.
Arrested around the same time as Anne was her brother George, musician Mark Smeaton, Henry Norris, William Brereton, and Francis Weston. The latter four were executed for having sexual relations with the queen, though most everyone agrees they were trumped up charges and she was never intimately with any of them. Having intimate relations with her brother was the icing on the cake for many of the day. As stated earlier, she and her brother were very close. As her enemies closed in around her before her arrest and she kept having miscarriages, her family became the only people she could rely on. Her father was all about saving his own skin, so it was her brother Anne clung to. Since she was already being charged with adultery, incest with her brother was just a small step. The "evidence" was testimony by various ladies-in-waiting and others who were close to Anne. This testimony has to be taken at face value, because no one wanted to be caught up with Anne and possibly executed. They would say anything to save their own necks. Add to that Anne and George were very religious. I don't believe either would do anything to risk their souls being condemned to hell.
There is one person that I am aware of who believes Anne and George had sexual relations. GW Bernard cites a poem (written by Lancelot de Carle) written shortly after her execution where a pregnant lady-in-waiting to Anne is accused by her (the L-i-W) brother of being promiscuous. The lady responds that she is by no means as bad as Anne and George. Bernard believes the lady to be the Countess of Worcester, who provided much "evidence" against Anne. Again, people will say anything to save their own necks, so I'm don't really buy into Bernard's interpretation of the poem.
I apologize for not having any in-text citations above--I'm at work on lunch break and all my books are conveniently at home. For those that are more interested, Alison Weir has written many books about the Tudors and a few about the Plantagenets. The best ones that include Anne would be "The Lady in the Tower: The Fall of Anne Boleyn" (2009), "Henry VIII: The King and His Court" (2001), and "The Six Wives of Henry VIII" (1991). I highly recommend her--easy to read and meticulously researched. Make sure you choose her nonfiction titles, as she has branched into fiction in the last 5 years or so.
Is something going on with Anne Boleyn in pop culture right now? This question was just asked a week ago.
This question was explored here:
http://www.reddit.com/r/AskHistorians/comments/261paz/how_historically_accurate_is_that_anne_boleyn/