I will focus specifically on the Moon here. I am not a geologist or selenologist, but am a space program historian who has done a fair bit of study related to the Moon (specifically in this case as it relates to the Apollo Program and the findings from those missions), so might be able to give some insight into your question.

When talking about small scale surface structures on our Moon (being the closest celestial body to us), prior to the 1960s, there was a tremendous amount of speculation and attempts at "best guessing" about what the surface would be like.

The idea of there possibly being small scale Earth-like stalagmite-like features rising from the lunar surface is a bit of a misnomer. As you pointed out, literal stalagmite and flowstone mound rock formation features (which are a type of "speleothems", or "cave formations") are built up over time in cave structures, created via liquefied mineral material drippings falling from above that accumulate beneath on the cave floor and solidify, eventually forming into those characteristic tall, thin mounds of mineral deposit rock. From a geological - or in the case of the Moon, a "selenological" perspective (selonology typically refers to the geologic/astronomic study of the Moon, as "geo" is related to Earth and "selene" is related to lunar) - it was well appreciated in the scientific community even prior to the Apollo era that literal lunar stalagmite features would not be present up there on the lunar surface due to lack of liquid water.

I think I know where you are coming from though when referring to early (prior to the mid-1960s) 20th century artwork that depicted what many thought the lunar surface may look like. Artwork like this from that era for example showed the idea of some small-scale lunar surface rock formations and larger mountain features, some of which having shape characteristics that appear somewhat similar to (though not thought even at that time to be related to) liquid-formed stalagmites. You see artwork from this pre-Apollo period by famous space artists like Chesley Bonestell for example which typically speculated that the lunar surface would be very "craggy", with hills and mountain features having lots of very jagged, vertical peaks. Some of this artistic interpretation was just pure romanticized landscaping, though it did seem to have some speculative scientific thought process behind it.

Of course, we quickly confirmed during the Apollo era, once we began sending missions to the more mountainous highland regions of the Moon in particular (Apollo 15, 16 and 17) that this was not the case at all.

Prior to the 1960s, remote sensing telescopic imagery and radar studies from Earth of the Moon simply lacked the resolution to accurately discern the true shape of the large scale mountainous features, and small scale features like rocks and boulders were not even close to being visually detectable from Earth. It really was not until proper remote sensing mapping satellites (like the five American Lunar Orbiter satellite camera & instrumentation platforms) were sent into lunar orbit in the mid-1960s during the run-up to Apollo to get a much closer look at our Moon that it started to become clear that the lunar mountains and large scale surface features did not typically appear as the sharp, jagged vertical alpine-style peaks the earlier artwork had depicted them to be, with the mountains of the Moon found to typically appear much smoother and flatter-topped, more like large mounds or dunes.

Once the Apollo crews got down there on the lunar surface and really got a close look at the vertical features, the lack of craggy, jagged mountains became much more apparent. Here for example is a picture from the lunar surface taken during the Apollo 17 mission to the Valley at Taurus Littrow, showing the northern slope of a large scale surface feature known as the "South Massif". This shot was taken through one of the lunar module windows shortly after the LM "Challenger" had landed there. That massif you see in that picture actually climbs about 7875 feet above the valley floor where the LM was, so that is not just some minor hill or mound. It really is a legitimate mountain. The "North Massif" on the other side of the Littrow valley at that same Apollo 17 landing site is even taller, climbing some 8700 feet above the valley floor, and we see it shares much the same shape characteristics as its southern partner massif, lacking the prior speculated sharp jagged alpine peak shapes that earlier lunar surface artwork depicted.

So, why were the lunar mountains and vertical rock features up there depicted as being sharp and jagged in the early artwork? Of course, due to limited resolution offered by terrestrial-based telescopes (and the rather low resolution radar studies done of the Moon from Earth) prior to the mid-1960s, these artists were forced to speculate about what the mountainous vertical features might look like up close, basing their guesses on our terrestrial understanding of geologic mountain formation, growth and shaping over many millions of years of time.

It was known well prior to the 1960s that the Moon had mountains of course, as shadow studies in particular, viewing the Moon at different sun illumination angles using telescopes on Earth, clearly showed there were vertical mountains rising off the lunar regolith. It was also known that the Moon lacked an appreciable atmosphere, so the idea of weathering effects like we have on Earth (wind and rain and water flow/frost for example), which can shape mountains and vertical rock features by slowly eroding them and "smoothing" their jagged features out, was not going to play a role in lunar mountain shaping, even over very long time frames. We still have many ranges on Earth with jagged mountain peaks, largely because although they are constantly being subjected to terrestrial weathering effects wearing them down, many are also still growing in size due to tectonic and volcanic activity, allowing those jagged peaked mountains, through growth, to somewhat keep up with the terrestrial weathering effects constantly battering them, helping them maintain their jagged shape.

Because of the known lack of wind and rain weathering effects or selenologically recent tectonic activity on the Moon though, many artists assumed lunar mountains as being initially formed as jagged and craggy, and figured these peaks would not have been subjected to any weather effect smoothing over the many millions of years of their existence like Earthly vertical geologic surface features must deal with.

What they were not taking into account in their speculative artwork however was what we now know to be the very powerful long term sculpting effects on the lunar surface from very consistent micro-meteoroid, asteroid, as well as cometary bombardment over the 4.5 or so billion years of the Moon's existence. With the Moon lacking an appreciable atmosphere to help deflect or burn up any of this incoming natural space debris, the lunar surface features are being consistently "weathered" by this high-speed deep space bombardment, ranging in size from tiny dust-sized particles to larger space rocks. These many countless fast moving impactors, as well as rock debris scattered from crater-forming meteorite strikes, over long geologic timeframes, have served to smooth out the lunar mountain features and "weather" away any jagged, craggy peaks that the early pre-Apollo era artists speculated we would see up there on the Moon. That leaves us with the much smoother-looking mountainous features dotting the lunar landscape that we today know are up there.