I'm coming at this question from the perspective of literary and cultural history.

So, Victorian attitudes about the Middle Ages stem from the mid-eighteenth century because of two popular cultural movements. First, the rise of Gothic literature full of creepy castles, ghosts, murdered family members, ruined abbeys, and beautiful heroines meant that anything vaguely medieval, however imaginary or fanciful, was a hot commodity, especially for "impressionable young ladies" of the literate middle and upper classes. These novels (by Ann Radcliffe, Matthew Lewis, Horace Walpole) often presented the medieval past as dark, sinister, full of temptation and violence, but also thrilling, exciting, and usually with a romance thrown in for good measure. Sometimes, it was incest with the devil in a monastery! (See Lewis's The Monk)

These novels built on stereotypes that persisted from the earlier eighteenth century that thought medieval people were crude, ignorant, despotic, and barbaric. The Enlightenment of the eighteenth century was the period of history that originated the idea of the "Dark Ages" in comparison to itself, since so many branches of knowledge were developing during this time.

Second, the 1740s and onward saw a huge architectural interest in Gothic architecture (once people realized that all these abbeys and churches across England were falling to pieces and giving fodder to the aforementioned authors). Perhaps because people wanted something different from the popular Neoclassical style, or because it was a bit closer to home, (rich) people began wanting more "medieval" details in their homes and public buildings. The mania for recreating Gothic architecture reached its zenith in the Victorian period, and some architects event went so far as to "re-Gothify" genuine Gothic churches to make them fit the idea of what they thought Gothic architecture "should" look like. Writers and activists like William Morris (more on him later) were infuriated by this.

The "time of glorious romantic chivalry" is also certainly true, and it was largely inspired by a confluence of ideas emerging in the mid-nineteenth century. For instance, John Ruskin's essay on Art Criticism, "On the Nature of Gothic" in his book The Stones of Venice described and valorized the art of Renaissance and Middle Ages as being neither something young women were into, nor a trifle for rich people, but an artistic style that could be looked to for the meaning of Art or Beauty. These ideas influenced the Pre-Raphaelites, whose artistic style and subject matter reflected what they thought was the spirit of the Middle Ages. So they painted a lot of pouting women with long, luscious hair in medieval dress, and a lot of people loved and were influenced by their art.

And people like William Morris, who founded the Arts and Crafts movement, came after Ruskin and promoted a return to medieval-style craftsmanship prior to industrialization--wood block printing, hand-carving, tapestry weaving, intricate embroidery, etc. Morris was a Socialist, so he promoted these crafts as a way to give factory workers a sense of purpose and pride in working. The idea was that medieval artisans had more satisfaction and made more beautiful things because they hadn't been numbed by industrialization. Basically, Morris and others thought that Medieval people knew a lot of things the Enlightenment didn't give them credit for.

Next, the rediscovery and publication of medieval manuscripts such as Beowulf in 1815 influenced poets like Alfred Lord Tennyson and the Pre-Raphaelites to focus on the medieval period. Tennyson translated the Old English poem "The Battle of Brunanburh" in a fit of medievalism, and his famous Idylls of the King (1859-1895) used the Arthurian legends as a way to reflect on Victorian morality, ideas about masculinity, death, etc. So while the stories themselves are set in the Middle Ages, they mix a lot of the stereotypes of chivalry with what Tennyson wished chivalry was in his own time. In the 1820s Sir Walter Scott's novel Ivanhoe touched on similar ideas about the Crusades. Although the published Beowulf and Ivanhoe aren't strictly Victorian, they had a lot of influence on Victorian culture.

Finally, there's the issue of religion. The Victorian period saw lots of religious diversity in England, from staunch Anglicans to dissenting Methodists, revivalists, immigrant Jews, and both born and converted Catholics (there are many more groups that I'm missing, of course). In the main, despite conflict, the Protestants stuck together against the Catholics and Jews. Thus, although they might admire and want to bring back the art or aesthetics of the Middle Ages, many Victorians were very suspicious of the religions of the Middle Ages. This also extends to Islam. The Ottoman empire fueled all kinds of exotic imaginings for the British Romantics and Victorians, but Islam itself was usually either considered irrelevant or pitted as a foil for Christianity.

tl;dr Most Victorians probably saw the Middle Ages as simultaneously an age of beauty, romance, and chivalry and as a period of despotism, ignorance, and fanaticism. Most Victorians probably got their ideas about the wonders of the Medieval period from art, literature, and architecture, as well as inherited prejudice about Medieval ignorance from the Enlightenment and Gothic novels.

Edit: minor grammar.