I'd recommend two sources that cover the reception with relation to the competition: Jeff Ryan's "How Nintendo Conquered America" and Blake Harris's more recent "Console Wars."

One of the things that Ryan focused on is Mario 64's innovation (indeed, Nintendo's promotional materials focused on this too) -- it' was one of the first games where you could control the camera in all directions. Mario 64 was one of the few games available at the console's launch, and it showcased the new console hardware and possibilities that weren't available on the Playstation or Saturn. I can't speak to the greatness it immediately received, but a lot of the early reviews certainly focused on its innovative features and newness that otherwise wasn't available on other consoles or, for that matter, Mario Games (while Mario RPG was 3d, this allowed complete control over his moves and attacks)

Well, this is something I never expected to see here, rifled in between people asking about Byzantine history, salt and the history of murder, etc. But, I have something you might be interested in.

If you want to hear some perspectives from probably one of their closest rivals on the mascot platformer genre, Naughty Dog (the developers of Crash Bandicoot), you can check out this blog for some insight: http://all-things-andy-gavin.com/2011/02/02/making-crash-bandicoot-part-1/

It is written by Andy Gavin, who was a co-founder of the Naughty Dog studio, who also worked as producer, lead programmer and designer on the Crash series. It is worth noting that since he is in the competition towards M64, of course his views will reflect that tension, so he might not be as glowing towards Mario as he is towards his own child, Crash. Over the course of the blog he brings in other minds from Naughty Dog to write some guest additions, and it is probably one of the more interesting dev diaries out there when it comes to this generation of games.

While at first it discusses the nature of making a mascot game, and the development of Crash Bandicoot 1 for the PS1, in later episodes of the blog they eventually go into discussing Mario 64 and the uncertainty on both sides towards how to take each other.

This is an interesting read just for the Crash Bandicoot stuff alone (such as how ND broke the rules to make the game). But around part 6 of the blog (http://all-things-andy-gavin.com/2011/02/07/making-crash-bandicoot-part-6/) he starts discussing the E3 demo for the release of Crash and Mario, and the two were somewhat going head-to-head on the stage. Included are small discussions of Miyamoto playing some Crash, and them playing some Mario, as well as lots of pictures of pre-development and some stuff from the E3 conference. He starts bringing on other developers in to discuss their views over the course of the blog too, such as Jason Rubin, another key name at Naughty Dog. It is an interesting look into development rivalry.

e: for the sake of it, I will directly quote Andy discussing the situation here incase anyone doesn't want to root through the blog to find it.

Mario was a bit of a two edged sword for us. First of all, the attention it garnered helped force us into the limelight. Sega was engaged in the slow process of killing themselves with bad decisions and bad products, and so Sony and Nintendo found themselves head to head. This literally put Crash and Mario into the ring together. In fact, this was depicted on the cover of at least one game magazine (along with Sonic who declined to enter the ring).

In any case, since Crash released about a month after Mario the press often assumed that we had copied various elements, which always bugged us to no end, as both games were developed with no real knowledge of each other. Crash was nearly beta by the time we saw Mario at E3, and gold mastered by the time the N64 shipped and we could play it. Both games took very different approaches to the then unproven 3D CAG genre.

With Crash we decided to emphasize detailed cartoon visuals and classic Donkey Kong Style gameplay. So we used a camera on rails (albeit branching rails).

With the N64’s VERY limited texture system and poly count, but with its smoothing and z-buffer, Mario chose to go with a very loosely defined polygonal free roaming world and a much more playground style of gameplay.

Personally when I first got my hands on Mario I was like WTF? How is anyone going to know what to do here? And although there was a pretty real sense of marvel in this funny new world, I never found it very fun. The early camera AI was brutally frustrating. And the Mario voiceover. I still cringe, “It’sa me, Mario!” Still the game was brilliantly innovative, although I remain convinced that if anyone but Miyamoto had made the game it would have flopped. Really, the future lay in the hybrid of the two.

Critics loved Mario. Perhaps because many of them were Nintendo fan boys, perhaps because it was more innovative (and it was). But the players loved both, because they sure bought a LOT of Crash Bandicoots too, approximately 35-40 million of our four PS1 games.

In a lot of ways Crash was the last of the great video game mascot characters, despite the fact that Sony never really wanted a mascot. We set out to fill this void, and made a game to do it, but we never really expected – only hoped – that it would happen. By the era of PS2 and X-box, the youthful generation of video game players had grown up, and the platforms began to appeal to a much wider age range. With this, and increased graphics horsepower that made possible more realistic games, came a shift to more mature subjects. The era of GTA, of Modern Warfare and Halo. Sophisticated and dark games mirroring R-rated action movies.

A part of me misses the simple, but highly crafted comic fun Crash represented.

Source: http://all-things-andy-gavin.com/2011/02/07/making-crash-bandicoot-part-6/ (there is more from Jason within this link, but that would make this post very long.)

I think one IGN reviewer put it best - "Overall, the biggest obstacle in the game is finding the correct viewpoint. The camera, which moves on its own (unless directed manually by using the four gold buttons), tries to find the optimal angle to view the action," 1 -

This was the biggest issue people encountered with the first 3D games, camera control. Today we take camera control for granted, and its usually something you don't pay much attention to. When 3D games games were first released, navigating and exploring the environment was a difficult process.

Satoru Iwata commenting on the free open 3D world said the following “Wandering all around provides a lot of freedom but carries with it the problem of not knowing where to go and getting lost." 2 Using an external camera to view action was a new concept for many players.

Another problem remarked on by a Nintendo Developer in a 2008 Nintendo Power interview, was player control, “Everyone has probably experienced how hard it can be to go where you want to go when moving your character around in 3-D space. When an opponent is approaching, in order to attack with your sword, you’ve got to position yourself in such a way as to hit it, and that can be quite difficult."

Another unintended consequence of bringing Super Mario into the next generation, 3D gameplay can introduce barriers to new players, Shigeru Miyamoto "By going into 3D, the games have become more complicated. Before that, the Mario games were the type of thing that anybody could pick up and play very easily. "

Overall critical reception was good, Graphics and Gameplay were rated Excellent in most major reviews, but controls and sound were both scored Fair or Good. Cartridge technology has a lot of shortcomings compared to the CD based systems like Playstation and Sega Saturn, both released before the N64. Sound samples made use of heavy compression to fit the limited storage capacity of cartridges. You won't find any N64 games with full orchestral CD quality soundtracks like those released on the Playstation during the same period. Overall critics scored gameplay high on Super Mario 64, and this is what has helped it retain popularity over many years. It took several years of development and testing by Nintendo EAD to get the controls right for Super Mario 64, which is a big part of why it is considered such a timeless classic.

Sources

1 - http://www.ign.com/articles/1996/09/26/super-mario-64

2 - https://dromble.wordpress.com/2013/09/21/nintendos-frustrations-with-3d-mario/