Every rebuttal mentions culture and ideology, but those don't just spontaneously appear. There has to be some causal series of events that brought forth said culture. Wouldn't the root of that causal chain be geography? Is my interpretation broader than the intended meaning of the term? Because otherwise I do not see how one could argue against the idea. Are all historians dualists? Where is the culture coming from if not environmental stimuli?
I had a similar conversation with a friend yesterday. I don't like to tie myself down to anthropological theory, but I'm certainly against Environmental Determinism (Diamond isn't purely an environmental determinist, but near enough that I'll be using be shorthanding his theory that way here). If you twist my arm, I'll generally side with Historical Particularism or at least something akin to that. It's anthropological theory for those who don't like anthropological theory!
The basics of Historical Particularism are these:
Other people will likely have their own frameworks to work with, but in general the thing that sets them apart from an Environmental Deterministic view is the value placed on human agency.
Diamond makes a big deal about the Spanish conquest of the Aztec Empire and the idea that environmental differences between Eurasia and the Americas determined the outcome of that event. But there are plenty of moments leading up to the Fall of Tenochtitlan where the decisions of individuals shaped the course of events. Years before Geronimo de Aguilar had been shipwrecked in the Yucatan. By the time Cortes arrived, Aguilar was one of two survivors. Luck spared him from diseases and the choices of his captors spared him from execution / sacrifice. His own choices led him into Cortes' service as a translator, while Gonzalo Guerrero (the other survivor) chose to ally himself with the Maya. Without Aguilar serving as the first link in the daisy chain of translations, Cortes' expedition would have been crippled. For Cortes himself, he was tasked with establishing a trading port on the coast of Mexico only; he chose to venture further and seek his fortune through conquest rather than trading. The Tlaxcala leadership chose to spare the Cortes and his men when they had them on the ropes, opting against the advise of their general, in order to turn the Spanish loose against their Aztec enemies. Ixtlilxochitl II chose to ally himself with the Spanish to settle an old grudge and overthrow his brother for control of Texcoco, removing one of the Mexica's primary allies from the picture before the siege of Tenochtitlan. I could go on, but you get the idea. History pivots on choices like these, and they're not subject to the predictions of environmental determinism.
Not an answer but an auxiliary question: has Diamond ever addressed any of the criticisms towards his work?