Hurray! My area of expertise.
To start in the arcades, as it was the first exposure for the general populous to this video game thing, arcade games were always seen largely as a novelty. A staple novelty, maybe a dangerous one, but always something with an appeal which was measured in terms of it’s newness rather than how long it stuck with you. After Space Invaders, the public perception of games could largely be put in two words: “Space” and “shoot”.
In the early 1980s, video games were a teenage pursuit. Even though the home consoles which existed did appeal rather heavily to younger demographics, that was not a focus of the marketing (which was very underdeveloped) or the media at large. Therefore the teenage mindset became associated with the arcade in all it’s forms. Delinquency charges followed arcade games all throughout the United States, Britain, and even in the Phillipines where they were banned for a time. On the positive end, they were seen as an introduction to computer technology which was fast becoming a big thing as well as an inherited non-violent outlet for aggressive emotions. The documentary Video Games: A Public Perspective from 1982 is a somewhat bought for but I believe honest portrayal of the older set’s view of the situation.
Now this does neglect the players themselves which were obviously the real core of the phenomenon. Why did they enjoy arcade games when that style is seen today as so passe? Not knowing any better I think was a minor factor, but it was definitely one. The people who played arcade games were not generally the type who played home computer games, which were often ‘deeper’ in their gameplay systems but lower on their action. Without a proper blending of action and depth which only appeared late in the initial period (games like Pitfall!), there was never going to be mainstream appeal.
The social aspect was also big and has really been at the center of video games being a part of mainstream culture. You point to 1998 as the video games having attained respect, but I would really push that to 2000 with the launch of the PS2 and the real emergence of multiplayer in PC games, though that’s obviously beyond the scope of this particular forum. Arcades drew in so many people because they were far more than a place to play games. There was a social aspect in the games themselves through a high score, and novelty is something that you can share.
The early 80s was also a time where a lot of newness was sought after due to the stagnation of mainstream culture which would be revived with the likes of MTV (regardless of my own feelings about that particular trend). Asteroids was a pulpy expansion on the thrill of navigating the Asteroid field in Star Wars which further entrenched young minds in sci-fi, for instance. Video games were also seen as a vessel of technology which were going to give improvements across every social area and there was an air of encouragement for young people to interact with technology. Those are the major pieces as to the how and why of the cultural impact at the beginning.
Skipping over all the factors of the crash (which I explained in another post on this forum), we get to the Nintendo-dominated late 80s. Obviously there are a lot of arcade ports to start and even Super Mario Bros displays some qualities of that quarter-munching action, but by the time Legend of Zelda and Dragon Quest hit it’s an entirely different ball game. The console games have taken on a quality far more comparable to that of computer games than arcade games in terms of their playstyle and nuance. Some of this was economical, to make games which could be justified at an over $50 (in 1980s money) price tag, but it was also driven by the artistic ambition of creators like Shigeru Miyamoto.
This is probably the point where the attitude between the general public and the players has the widest gap. Because of Nintendo’s focus of marketing on adolescent boys, the public view of video games became more toy-like than novelty. That’s still in a way not seeing a lot of intrinsic value in the product, but extrinsically it was seen as something they could manipulate rather than just experience. Consequentially, the content of games was much more dutifully scrutinized. Inappropriate content for a 6-12 bracket was not going to fly under Nintendo’s watch and that put games into a narrow box of what was acceptable.
With these games becoming more focused on time investment, you also start to see the first real complaints about addiction cropping up. There had been some talk of overzealous playing during the arcade era but nothing to the level of what was thrown at Nintendo. A lot of parent support groups tried to tag Nintendo with destroying America’s youth or whatever, though this criticism actually has some degree of understanding to it. This tied into the refined rewards structures that Mega Man 2 could create (along with ill-advanced technology which didn’t allow you to save). With all this, video games also got tagged with “a waste of time”, especially for anyone above that age range as mentioned before.
I think the concept of “levels” was a pretty big player here. Arcade games very rarely had a sense of advancement or content which visibly changed from one area to the next. Levels as self-contained spaces which brought new elements to the gameplay, increased the challenge, and provided a sense of space really didn’t exist before these deeper console games took hold. They could take advantage of technology to bring artistic visions to life in far greater detail than most home computer games could, largely helped by the advancements made by the cartridge format over the years (examples like Super Mario Bros 3 adding in extra chips for memory and diagonal scrolling).