Today, when someone mentions a "spooky" or "scary" looking house one often imagines an old Victorian house with wrought iron fencing, overgrown lawns, etc. A google image search of "haunted house" turns up many, many "scary" Victorian houses.
Did people in the English speaking world in the Victorian era look at, say, old decrepit Georgian style buildings as being scary in the same we see old, ill-kept Victorian buildings?
Decay and decrepitude are kind of universal - I can give a couple of examples from opposite ends of the Victorian era (1837-1901):
I looked upon the scene before me—upon the mere house, and the simple landscape features of the domain—upon the bleak walls—upon the vacant eye-like windows—upon a few rank sedges—and upon a few white trunks of decayed trees—with an utter depression of soul which I can compare to no earthly sensation more properly than to the after-dream of the reveller upon opium—the bitter lapse into every-day life—the hideous dropping off of the veil. There was an iciness, a sinking, a sickening of the heart—an unredeemed dreariness of thought which no goading of the imagination could torture into aught of the sublime. What was it—I paused to think—what was it that so unnerved me in the contemplation of the House of Usher? It was a mystery all insoluble; nor could I grapple with the shadowy fancies that crowded upon me as I pondered. I was forced to fall back upon the unsatisfactory conclusion, that while, beyond doubt, there are combinations of very simple natural objects which have the power of thus affecting us, still the analysis of this power lies among considerations beyond our depth. It was possible, I reflected, that a mere different arrangement of the particulars of the scene, of the details of the picture, would be sufficient to modify, or perhaps to annihilate its capacity for sorrowful impression; and, acting upon this idea, I reined my horse to the precipitous brink of a black and lurid tarn that lay in unruffled lustre by the dwelling, and gazed down—but with a shudder even more thrilling than before—upon the remodelled and inverted images of the gray sedge, and the ghastly tree-stems, and the vacant and eye-like windows.
At one corner of the room was a heavy door. I tried it, for, since I could not find the key of the room or the key of the outer door, which was the main object of my search, I must make further examination, or all my efforts would be in vain. It was open, and led through a stone passage to a circular stairway, which went steeply down. I descended, minding carefully where I went, for the stairs were dark, being only lit by loopholes in the heavy masonry. At the bottom there was a dark, tunnel-like passage, through which came a deathly, sickly odour, the odour of old earth newly turned. As I went through the passage the smell grew closer and heavier. At last I pulled open a heavy door which stood ajar, and found myself in an old, ruined chapel, which had evidently been used as a graveyard. The roof was broken, and in two places were steps leading to vaults, but the ground had recently been dug over, and the earth placed in great wooden boxes, manifestly those which had been brought by the Slovaks. There was nobody about, and I made search for any further outlet, but there was none. Then I went over every inch of the ground, so as not to lose a chance. I went down even into the vaults, where the dim light struggled, although to do so was a dread to my very soul. Into two of these I went, but saw nothing except fragments of old coffins and piles of dust; in the third, however, I made a discovery.
The Victorian horror aesthetic in Gothic fiction and ghost stories tended to mortuary architecture (tombs, cemeteries, etc.), and ruins - which could be alternately romantic or terrifying depending on the moods - but they were, just as us, looking backwards on the buildings and structures that were old to them.
This has to be contrasted with a couple other developments: the emphasis on the horror of nature (essentially, the lack of architecture), as exemplified by Algernon Blackwood's "The Wendigo" (1910), and the horror of new technology and industrial architecture, which could be characterized as inhuman or monstrous. I recall one scene from the Nautilus, which Verne was careful to initially describe as a horrendous monster, but then revealed as a miracle of modern technology...and yet:
For two hours Ned’s anger grew fiercer. He shouted and shrieked in vain. The metal walls were deaf. I could not hear a single sound in the boat, which lay as if dead. It was not moving, for I would have felt the trembling of the hull from the throbbing of the screw. It was probably plunged into the depths of the waters, far from the earth. The deathly silence felt terrifying.