There is plenty of circumstantial evidence that fed contemporary rumours and which make it possible for historical fiction writers to play with the idea, but this is one of those issues that can't really be proven one way or the other unless someone finds some new and incriminating piece of evidence lurking in an archive or attic.

There were a lot of reasons for the (mostly) protestant Scottish lords to dislike Rizzio (or Riccio): he was Catholic, he was foreign and he was a social climber from a poor family. Mary liked his voice and heaped privileges on him. She made him her gentleman of the privy chamber and there were lots of high ranking officials who would have resented the fact that he was taking a position that ought to be given to a protestant Scot of noble birth. Eventually, she made him her French secretary. Both of these positions put him in close proximity to Mary out of necessity, although since she liked his singing voice, he was probably in her company more than even the secretarial duties required.

Most primary source materials that mention a possible affair cite circumstantial evidence: they stayed up late playing cards together, she happened to get pregnant with James VI/I while Rizzio was in her employ, people felt that Rizzio dressed and behaved above his origins and station, there are instances of people making off-hand comments alluding to James VI/I's "dubious" parentage. Even after he was murdered, Mary was so distraught over it that some took that as confirmation of a relationship.

In more casual terms, important and high-ranking members of the Scottish nobility didn't like him and they talked a lot of shit about him. They felt he was not good counsel for Mary, they resented his quick rise and he was a Catholic foreigner to boot so a lot of individuals wanted him gone. It didn't help that Mary's husband Lord Darnley was a jealous and insecure man who resented being king consort rather than a sovereign ruler of Scotland through marriage. It's very likely that Rizzio's enemies played on that insecurity to turn him against the man.

Whether Rizzio and Mary were ever more than close friends is something we'll probably never know. In early modern Scotland, the best way to discredit someone was by accusing them of sexual and moral indiscretions - there were also rumours that Rizzio was a necromancer and a papal spy/secret envoy. When we take all that into account, it really does look more and more like the lords who wanted him gone were just throwing a bunch of nasty rumours around hoping something would stick.

Sources

Most academic biographies of Mary, Queen of Scots engage with this issue to some degree.

Steuart, A. F. Seigneur Davie: a Sketch Life of David Riccio (Rizzio) London: Sands, 1922.

If you would like some primary source citations, I can track them down, but I'm going on memory for some of the examples.

Edit: a word