Manliness and masculinity were important to the Victorians, but the concept went through several changes.
Edit: Or rather, the emphasis on various elements of it.
I don't believe I can answer this question fully, but I can certainly address one aspect of it.
There was a growing fear of degeneration during the Victorian Era, and this fear went hand in hand with the idea that men were becoming weaker, or more effeminate.
That idea then also contributed to the fear of the 'New Woman', who was a more masculine, independent, educated woman. She was often typified as wearing ties, riding suits, and cycling! The New Woman is more of a stereotype, though, but suffragettes certainly did exist, and the idea of women gaining more power often seemed like a threat to men.
As Sarah Grand wrote in her essay, ‘The New Aspect of the Woman Question’:
Man is… apt to accuse us of wishing to ape him. Of old if a woman ventured to be at all unconventional, man was allowed to slander her with the imputation that she must be abandoned, and he really believed it because with him liberty meant license. He has never accused us of trying to emulate him in any noble, manly quality, because the cultivation of noble qualities has not hitherto been a favourite pursuit of his…
From: The North American Review, 158.448 (1894), pp. 270-276 (p. 273).
Edit: formatting.