Novels were particularly associated with frivolous female readers. The fiction-reading public of the eighteenth and early nineteenth centuries was predominantly female, so there was some truth to this stereotype, but the characterization very much related to general societal prejudices about women. It's somewhat recursive: women are frivolous and mostly interested in silly things, so if they like novels, novels must be silly; novels are made-up stories that rarely reflect real life, and since women tend to like them, that just shows that women can't tell how unrealistic they are or aren't discerning enough to care.
“Oh! No, I only mean what I have read about. It always puts me in mind of the country that Emily and her father travelled through, in The Mysteries of Udolpho. But you never read novels, I dare say?”
“Why not?”
“Because they are not clever enough for you—gentlemen read better books.”
“The person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid. I have read all Mrs. Radcliffe’s works, and most of them with great pleasure. The Mysteries of Udolpho, when I had once begun it, I could not lay down again; I remember finishing it in two days—my hair standing on end the whole time.”
“Yes,” added Miss Tilney, “and I remember that you undertook to read it aloud to me, and that when I was called away for only five minutes to answer a note, instead of waiting for me, you took the volume into the Hermitage Walk, and I was obliged to stay till you had finished it.”
“Thank you, Eleanor—a most honourable testimony. You see, Miss Morland, the injustice of your suspicions. Here was I, in my eagerness to get on, refusing to wait only five minutes for my sister, breaking the promise I had made of reading it aloud, and keeping her in suspense at a most interesting part, by running away with the volume, which, you are to observe, was her own, particularly her own. I am proud when I reflect on it, and I think it must establish me in your good opinion.”
“I am very glad to hear it indeed, and now I shall never be ashamed of liking Udolpho myself. But I really thought before, young men despised novels amazingly.”
“It is amazingly; it may well suggest amazement if they do—for they read nearly as many as women. I myself have read hundreds and hundreds. Do not imagine that you can cope with me in a knowledge of Julias and Louisas. If we proceed to particulars, and engage in the never-ceasing inquiry of ‘Have you read this?’ and ‘Have you read that?’ I shall soon leave you as far behind me as—what shall I say?—I want an appropriate simile.—as far as your friend Emily herself left poor Valancourt when she went with her aunt into Italy. Consider how many years I have had the start of you. I had entered on my studies at Oxford, while you were a good little girl working your sampler at home!”
(From Northanger Abbey, a little bit of an Author Filibuster in defense of novels.)
However, there were plenty of men who read novels as well. Literary critics were male, and they read novels for review - which could make or break them. (They were also often novelists or playwrights themselves.) Fanny Burney's father was part of a male literary circle, and she prepared her novel Evelina (1778) specifically to meet with their approval (through giving the heroine ample moral instruction from male characters), in order to benefit from the boost in sales and respectability - and while Austen likely didn't do the same, she did also benefit from male readers who wrote reviews.
One of the main criteria applied to novels by critics was whether or not moral instruction could be derived from it, and while Austen's characters lack the all-good/all-bad dichotomy of most didactic novels, this did work out in her favor: her books do teach the value of looking beyond first impressions, behaving respectably and cautiously with men, being humble, etc. Beyond that, they would ask whether it were amusing (yes) and/or realistic (yes), and about the quality/novelty of the story and writing style. We are today encouraged to read Austen through the lenses of romance and feminism, but to a period eye Sense and Sensibility was quite easily and acceptably seen as a critique of Marianne's damaging sensibility held up against Elinor's rationality and reserve. Likewise, Pride and Prejudice was deemed good primarily because of what Elizabeth Bennet teaches the reader, although her sparkling personality was attractive even then, with reviewers drawing comparisons even to Beatrice of Much Ado About Nothing - a massive compliment. (Oddly, there are no contemporary reviews of Mansfield Park, which is especially strange as this is easily the most didactic of all the novels.) As the books were published, they received more and more notice and were presented more and more seriously by the literary magazines. And, pursuant to your question, even private responses preserved in letters and diaries show a roughly gender-equal readership. (I suppose it must be mentioned that the Prince Regent read her work; he had his librarian give her a tour of Carlton House and ask that she dedicate her next book, which would be Emma, to him.)
And outside literary circles, too, Austen's novels were quite popular! Sense and Sensibility, Austen's first published novel, had an initial print run of between 750 and 1,000 copies, which was higher than the norm (500-750 copies) - and, remarkably, it sold out in about two years and was, again remarkably, reprinted. The book's popularity led to Pride and Prejudice having a first print run of 1,250-1,500 copies, and that sold out before the end of its first year, and the same was true of Mansfield Park. Emma had an even larger first edition of 2,000 copies, and sold out in less than a year. Northanger Abbey and Persuasion were published together in four volumes after Austen's death, with an initial print run of 1,750 copies, which did not sell out as quickly as the earlier ones. So I think it is very fair to say that Austen was extremely popular in her own time, despite her anonymity and the fact that she was a woman writing in a women's genre!