There are two reasons why Mexico wanted the Nueces to be the border - once which is clear (which I will call sovereignty) and one which is a bit more subtle (extended geography).

Sovereignty

Naturally, Mexico would wish to limits its territorial losses as much as possible. If you look at a map of the 19th century Mexican state of Coahuila y Tejas, Mexican Texas is has a rather boxy shape and is internally divided from Coahuila at the Nueces River. The Mexican argument was that even when Santa Ana conceded Texas independence, this would only apply to the territory of Mexican Texas, not to expansionist claims into Coahuila or other states.

Additionally, Mexican politicians and writers considered the loss of Texas to be a slight on national honor for which they especially blamed the United States. Not only did the Texan colonists migrate from the United States, but when the Independence War erupted some Americans outright travelled to Texas to fight against Mexico (the most famous example is Davey Crockett). Several years before Texas revolted, Mexican general Manuel Mier y Terán described Texan colonists as the forerunners of US territorial acquisition:

The department of Texas is contiguous to the most avid nation in the world. The North Americans have conquered whatever territory adjoins them

They begin by assuming rights, as in Texas, which it is impossible to sustain in a serious discussion . . .

In the meantime, the territory against which these machinations are directed, and which has usually remained unsettled, begins to be visited by adventurers and empresarios . . .

Shortly, some of these forerunners develop an interest which complicates the political administration of the territory; complaints, even threats, begin to be heard . . .

Having arrived at this stage - which is precisely that of Texas at this moment - diplomatic maneuvers begin. They incite uprisings in the territory in question . . .

After the Texas War for Independence, Mexico City newspaper El Mosquito Mexicano published an article in 1836 which opened with:

It would be most pleasing if the supreme government could achieve the nation's vengeance for the insults that have been waged against it and that it has suffered from that corrupt and aggressive part of North America

Finally, immediately before the Mexican-American War itself broke out, the Mexican general Pedro de Ampudia wrote to American general Zachary Taylor:

To explain to you the many grounds for just grievances felt by the Mexican nation, caused by the United States government, would be a loss of time . . . Your government . . .has exasperated the Mexican nation, bearing its conquering banner to the left bank of the Rio Bravo del Norte . . . I require you . . . to break up your camp and retire to the other bank of the Nueces River . . .

What comes to the fore from these excerpts is that Mexican leaders regarded the Nueces as the proper border of Texas, and that Mexican leaders regarded advocating for that border as an issue of national pride vis-a-vis US expansionism.

Extended Geography

I think it is important to clarify that the Rio Grande border was much more expansive than it might appear at first glance.

To go to a period map of Coahuila y Texas , Texas is a rather boxy area much smaller than it is today. The Coahuila half of the state plus Tamaulipas filled much of present-day southern Texas, Chihuahua was to the southwest, and a much larger New Mexico was to the west. In particular, notice that the Nueces itself is a relatively short river of only about 300 miles. A Texas with the Nueces at the border is relatively small and is contained within the territory of Mexican Texas.

When the Republic of Texas (and later the United States) claimed the Rio Grande as the border, they were NOT claiming this as the border just in southeast Texas where the colonists settled. They claimed the Rio Grande as the border for the entire length of the river all the way up to the headwaters. If you look at this 1837 map of Texas' claims, you will see that Texas claimed an enormous territory far larger than even present-day Texas. Among these claims were about half of New Mexico including Albuquerque and Santa Fe, a strip of central Colorado, a corner of Kansas, and even a bit of Wyoming.

If Mexico were to recognize the Rio Grande as the border, then that would entail ceding a vast swathe of land from several Mexican states across nearly 2000 miles of its northern frontier. To a Mexican leadership that not only smarted from losing Texas but also suspected further United States territorial ambitions, such a cession was deeply unappealing.

Sources

Chávez, Ernesto. The U.S. War with Mexico: A Brief History with Documents. Bedford /St. Martin's, 2008.

Henderson, Timothy J. A Glorious Defeat. Hill and Wang, 2008.

McConnell, James. “The Republic of Texas and the United States in 1837 | Library of Congress.” Library of Congress, https://loc.gov/resource/g3701sm.gct00482/?sp=26.

McKeehan, Wallace. “Coahuila y Tejas 1836 & Neighbors Tamaulipas & Nuevo Leon.” Sonsofdewittcolony.org, Sons of Dewitt Colony Texas, 1997, http://www.sonsofdewittcolony.org/co&tex1836.htm.

Texas General Land Office. “Mapping Texas - Coahuila y Texas: A Meeting Place.” Texas General Land Office, Medium, 30 Apr. 2020, https://txglo.medium.com/mapping-texas-coahuila-y-texas-a-meeting-place-2f3301d852c1.