The British public knew about Victoria and her eventual succession many years before Victoria herself did. Though you have to remember that for about a decade after her birth, there was still some hope that one of the royal princes would have a son that would be able to supplant the young girl in the line of succession. (The Duke of Cumberland did have a son, in fact, but as he was a younger uncle than her father the Duke of Kent, Victoria stayed heiress presumptive.)
This was one of many problems with the Kensington System, it kept Victoria in isolation not only from the public who wanted to see her but also from her royal family who presumably, would have brought her up in an environment and education befitting her status. Her mother, the Duchess of Kent, famously relied heavily on Sir John Conroy’s advice, an Irishman who some believe was in a sexual relationship with the widowed duchess. They were terrified that the royal family would take away her daughter, or worse, send someone to kill the young girl. So public viewings of a young Victoria were few and far between. When she played in the gardens at Kensington, people would line the fences trying to catch a glimpse. (Never mind the fact that one of the reasons they kept her away from the royal family was their terrible reputation with the British public. Victoria was presented as pure and angelic, and in Conroy/Kent’s opinions, any contact with her uncles would lose her the support of the public.)
When Victoria did ascend the throne, there were numerous worries, not the least of which was that the Kensington System had kept her isolated from her subjects. By it also had effectively kept her isolated from royal life and under the thumb of her mother and Conroy who were pushing for a regency. And, ultimately, the idea of a young woman as Britain’s Queen was highly irregular. (A bit of a laughable thought today, after 60+ years of QEII, but you have to remember that this was the 1830s and the last Queen had been Anne, who came to the throne unusually and reigned erratically.) So there were definitely widespread worries that Victoria was not up to snuff as a monarch. I’m not sure it’s possible to separate those fears from one another, to say that it was entirely her childhood isolation or the culturally acceptable sexism/ageism that caused the bulk of the fear, but it was there.
(Also, the previous heiress presumptive Charlotte had led a relatively public life, and died unexpectedly in childbirth — that also contributed to people being overly protective of the young Victoria.)
An interesting note to remember is that a rumor spread not long before she took the throne: Victoria had been almost deathly ill (with diphtheria, IIRC? It was typhoid fever.) and Conroy and the duchess had tried to use the situation and her illness to their advantage and force her to sign paperwork setting up a regency with themselves in charge. The young, very sick woman resisted every attempt, and ultimately pulled through the sickness. The story made the public gossip mill and was held up as an example by Victoria’s supporters that if/when she came to the throne, she would not allow a Conroy regency.
Sources:
Becoming Queen Victoria, by Kate Williams ISBN 978-0-345-47239-7
Longford, Elizabeth (2004). "Conroy, Sir John Ponsonby, first baronet (1786–1854), courtier". Oxford Dictionary of National Biography. doi:10.1093/ref:odnb/37309
Edit: My apologies for any typos, I did this on my phone.
Related question(s): What is the Kensington System?. Was it customary for royal heirs to be treated that way? If not, why was She different?