First, a few caveats to your assumption. For one thing, a lot more of the silver that ended up in China came not from Latin America but from Japan instead (although a lot of it did come either through Manila or around the cape). Secondly, although there was a reversal of the silver trade in the 1800s, with the Opium Wars and the Taiping Rebellion, China's silver demand resumed shortly thereafter (the "Mexican Dollar" essentially being the operative trade currency of China, after the Mexican war of independence).

The problem with the silver lies primarily in the form of the Great Depression. The reliance of many countries on the Gold Standard turned into disaster when in the face of the massive financial shock in 1929, the inability of countries to provide economic stimulus through monetary policy because of the gold standard caused some countries to abandon it entirely. This in turn would force other countries to abandon the gold standard, and so on, creating a cascading financial crisis that did much economic damage. The US was no exception to this, with severe restrictions on the gold trade coming on in 1933 and the Gold Reserve Act of 1934 allowing the government to devalue the gold dollar, increasing the money supply.

You may ask why this would affect China, as China was a country that was essentially run on a silver standard, rather than a gold one. Indeed, China managed to ride out the early stages of the Great Depression without much incident. Though the Qing Dynasty had long fallen and the nascent Republic of China and its instable, authoritarian Kuomintang/Guomindang (KMT) government had risen in its place, the silver situation had not changed too much aside from being formalized by the KMT. However, things would change, thanks to the 1934 Silver Purchase Act.

Silver miners in the US were a powerful industry force, and they pushed the US government to boost silver demand by buying silver at a higher price than current market prices. Led by Key Pittman, a senator from the silver producing state of Nevada, they pushed the US government to pass the Silver Purchase Act of 1934, which caused the US government to purchase silver from the public at a rate substantially higher than the market price.

Naturally, speculators with at least two brain cells quickly realized that there was an opportunity to make money through arbitrage. They bought lots and lots of silver, especially from China, in order to sell it to the US government at a higher price. This can be seen from inventories of Chinese banks, especially foreign ones.

The Chinese government, to their credit, had already seen this outcome when the legislation was first proposed. They failed to block the legislation despite lobbying, however, and now the Chinese were facing a currency crisis as all the silver was now being drained out of China. In an effort to prevent total financial collapse, China forced itself off the silver standard in 1935, and attempted to back their new fiat currency by purchasing foreign currency to use as reserves. However, this forced them to sell even more silver to do so, and the outbreak of the Second Sino-Japanese war in 1937 certainly did no favors to the Chinese government's ability to conduct fiscal and monetary policy. Though it triumphed over the Japanese, its monetary situation was in dire straits, and a last ditch attempt to reform the currency failed in 1947, an outcome which Milton Friedman argued led to the fall of the KMT government in mainland China and the rise of the CCP.

Sources: Franklin D. Roosevelt, Silver, and China, Friedman, Milton, JPE, 1992.
The Shanghai Capitalists and the Nationalist Government, 1927-1937. Coble, Parks, 1986.