This is a very interesting question that I've asked myself many times! My field is one that people frequently cite when professing a love for the Victorian era, as you point out, and lately I've been trying to get a more academic handle on the phenomenon.

Lack of historical knowledge can be one factor. For many, "Victorian" is used as a sort of catch-all term for the more recent past, including the Edwardian era and often bleeding back into the Georgian as well (at least the nineteenth century part - the "long Regency" - but often some of the late eighteenth, too). The number of analyses of 1920s fashion I've seen that act as though we went directly from the Victorian era to that decade! Others simply only know about the Victorian period, and might love aspects of the Georgian or Stuart and Tudor periods if they had more familiarity with them. We certainly read more actual Victorian fiction/see adaptations of Victorian fiction/read or see contemporary fiction set in that period than others, I think.

"Victorian" has come to mean a lot of things beyond "relating to the reign of Victoria, 1837-1901": broadly speaking, things that are recognizably not "modern", the way that we see ourselves or would like to be seen. This started relatively soon after the end of the queen's reign, with those in the late 1910s and 1920s seeing themselves as part of a "post-Victorian" age that was starkly different from what had come before. This has largely been studied in the context of early twentieth century writers like those of the Bloomsbury Group - Virginia Woolf, Vita Sackville-West, T. S. Eliot, et al. - who distinguished themselves or who were distinguished by others as being intentionally anti-Victorian, or whose pro-Victorian sentiments need to be examined beyond simply being considered conservatism. These writers generally came of age around the turn of the century, spent their childhoods in middle- or upper-class Victorian households, and went on to become an educated, elite group that conspicuously spurned the mores and standards of their parents' generation, which to them represented the whole of the Victorian era despite being a highly skewed view of the entire period and all of the people in it.

Twentieth century views of the era conformed more and more to that of the Bloomsbury Group as "moderns" opposed themselves to the Victorians, who were now the Other. They covered up piano legs for modesty (they didn't: that was a joke printed in 1839 about specifically American squeamishness that somehow got legs (please excuse the pun but I couldn't help myself)), we wear skirts that come to just below the knee or mid-calf. They forced women into horrible, tight corsets, we wear elasticated girdles and/or diet to stay in shape. They kept women oppressed and immured in the house, we totally allow them to work sort of. Flaws of current times were smoothly papered over or not actually noticed at all, while those of the past, particularly those that had been highlighted in nineteenth century fiction, were magnified and presented as relics. In 1948, a reviewer for the London Standard astutely noted:

Sustained by countless novels and plays and eagerly seized on by the makers of films is the Great Victorian Myth. The Myth has almost succeeded in converting the younger generation to a belief that Victorian domestic life was full of thwarted motherhood, tyrannical husbands and fathers, and spiritual frustration in dark, rambling houses. ... Clearly the thing to do was to torture the heroine, and if you put her in crinolines you could torture her much more severely.

At the same time, there was a nostalgia for the vanished past, and at the same time, I would argue, a continuation of the Othering that had appeared in the 1910s-1920s. It's probably easiest to point to that associated with the "Gay Nineties", which I wrote about previously here. It's also easy to see a similar longing for a supposed simpler time in the response to Gone with the Wind (published 1936, screened 1939) and in southern American portrayals of the antebellum, slaveholding period in general, including those still on view in plantation museums today.

Because of that intense negative and positive Othering, the period has retained a certain sense of importance. You have to have an opinion on it, whether it's positive ("I love the clothing!" "People were so refined back then") or negative ("They were oppressive"). We immediately get certain images in our heads when we see the word "Victorian", whether those images are inaccurate or accurate, and we react based on them.