At what level?

Prior to the modern era because of it's scarcity and cost to produce metal seldom was used beyond ornamental and otherwise minor function.

Basically until the development of the Bessemer Process made steel manufacturing fairly cheap, and then someone else realized that between steel reinforced concrete and steel frame construction, they could build very tall buildings if the frame was strong enough, buildings used wood or some form of stone (granite, fired clay, ect) as the frame of the building.

At the same time though, looking at something like Versailles you can't exactly say, "nope, no metal here." with a straight face. And even if it's just account for screws and nails, you still got some metal being used as a building material.

In that sense, rivets are ancient. Forged nails were used by ancient Egyptians in 3400 BCE. Screws as a fastener date back to roughly the first century.

But to answer your question, it really was about cost. By the time of the Crusades a knight could spend roughly what a house costs today just to get a well fitted suit of armor. Metal, particularly metal that wouldn't just rust or corrode, was extremely expensive, yet on the other hand stone and wood were common and more than capable. Basically until the later 1800's metal was a decoration and a fastener.