1961 appears to be close enough to still have the terrors of World War 2 in mind, so it seems unlikely there would have been another war if either Western Germany or “an anonymous actor” had taken down the Wall.
Also, I can’t imagine them wanting to preserve diplomatic relationships because that premise would go against general policy of that time (such as the Hallstein doctrine)...
Of course, the DDR could have always just built another wall if the first one had been torn down but at least that would’ve given people the chance to pack their things and decide which side of Berlin they wanted to stay on, and to not be separated from their loved ones, right?
So why did the Berlin Wall stay put?
The Wall was intentionally built inside the Soviet-occupied sector of Berlin. In some places it was close to the border but in others it was a fair distance from the demarcation. Any attempt to tear down the Wall would therefore have been a direct violation of the post-war agreements between the Allies that guaranteed each country control over their designated sector.
Prior to the construction of the wall, Soviet Premier Nikita Khrushchev had attempted to pressure the US, UK and France into withdrawing from their occupied sectors in Berlin. The Soviets were clearly ready to abandon the post-war agreements, but the US and Britain in particular explicitly refused to do so, insisting that the accords still held, in large part due to the fact that their withdrawal would leave West Berlin (an island within an East German sea) isolated from the West and would inevitably lead to East Germany assuming control over the entire city.
Given their insistence that the agreements still held in 1958, the US could not therefore do much about a wall that was built within the Soviet-occupied sector in 1961. While they may object to the presence of the wall, they acknowledged that the Soviets were within their rights to build it.