There wasn't really a single, universally accepted, institutionalized theory explaining continental deformation and orogeny (mountain building) before plate tectonics. I'll start with late 19th century theories as I'm not particularly well versed on prior geology.

Jean Baptiste Elie De Baumont proposed that the cold crust overlies a hot, but cooling interior. As the interior cools contracts, the crust does not and folds to take up the slack, wrinkling similar to the skin on a drying apple, creating ridges. Later, influential Austrian geologist Edward Suess expanded on this. He proposed that the oceans and continents were interchangeable. As the earth cools, previously continental areas collapse. The oceans retreat into these new basins, leaving some old oceanic areas high and dry and preserving marine fossils on dry land. This would sometimes reverse, with sea levels rising from sediment filling ocean basins. He though Europe had been formed by a series of contracting mountain building events, one of them forming the Alps. He greatly influenced Marcel Bertrand, who was at one time a student of Elie de Beaumont. Bertrand focused more on the mountains themselves than Suess. He rationalized and organized his predecessor's mountain building events and developed and expanded theories of overthrusting and folding of older marine sediments over younger ones. He thought the Alps had been formed by four separate, but related orogenic episodes to form one chain.

Geosynclines were popularized in the United States by James Hall and James Dana in the 1870s and later expanded by Emile Haug. Hall had proposed that oceanic crust alongside continents, are piled with sedment and sink the crust due to their weight. As the trough subsides into hotter material, the heat and pressure metamorphose the sediment and intrude plutons and volcanoes into the trough, uplifting it, and any marine fossils in it, into mountains. Dana believed the troughs were the result of already extant basins, caused by contraction, that filled with sediment due to gravity. He viewed oceans and continents as permanent features, in opposition to Suess's collapsing continents, and placed more emphasis on rising mountains than sinking seas.

The discovery of radioactivity as a constant source of heat in earth's interior and the development of isostasy in differentiating oceanic and continental crust, made contraction theory obsolete by the 1920s. Earlier studies of folding in the Alps and Appalachians that required much more horizontal movement than Suess or Bertrand's theories would allow for, confirmed this. Wegener introduced his theory of continental drift in 1912, which although lacking a convincing mechanism, still poked holes in much of accepted early 20th century geology. Reginald Daly in the US, and Arthur Holmes in the UK, combined Wegener's new theory of moving continents with the old geosynclinal theory.

Seismological studies had suggested that the earth was divided into three layers, crust, substrate, and core. Daly theorized that the substrate, or mantle, might be plastic enough to allow the crust to move horizontally it. He thought that enough sediment could fill geosyncline to rupture it, causing part of it to slide into the mantle, horizontally compressing and folding the sediments into mountains and pulling continents with them, causing them to drift.

Holmes went further with the plastic mantle and though it was hot and ductile enough to convect. Holmes believed that mid-ocean ridges were continental fragments left over from the break, with new oceanic crust formed on each side from upwelling mantle rock. This convection in turn drove downwarping at the opposite end of the continent from the ridge, forming geosynclines.

Holmes and Daly had something vaguely similar to plate tectonics, but their theories, along with continental drift in general, were never the majority view. They had mild support in Europe and hardly any in North America.

Mainstream geosyncline theory and its mechanisms were described as pretty vague by the plate tectonics pioneers who were taught the theory in college, when continents were still thought to be fixed, but were probably the most popular explanation for mountain building before the 1960s, especially in the U.S and Canada.

Sources:

Naomi Oreskes Plate Tectonics: An Insider's History of the Modern Theory of the Earth

Gabriel Gohau A History of Geology

Naomi Oreskes The Rejection of Continental Drift: Theory and Method in American Earth Science