As the title says, I am looking for primary sources that argue about who should get to vote.
Mill in the 1800s I know was arguing for why women should be able to vote in a robust way. I assume that he has detractors that argue in the same way. Even if there aren't as many academics like mill taking on the he question, I would imagine we could see opinion peices and the like.
If you're looking for examples of popular sentiment on those sorts of issues a good bet would be newspapers around the time the franchise was getting extended to one group or another. Opinion and letters pages could get pretty incandescent on certain topics, and more typical articles on the subjects of the day are usually filled with quotes from relevant individuals.
Most major papers will have some way of getting at their full archives; they're usually paywalled behind a subscription package, but a reasonably well-off library should have a few of those for visitors who want to go digging. The NYT archive in particular is really search-friendly compared to a lot of papers, and is one of the few I've seen where full-text searches are pretty easy.