If you mean with "big," internationally known, translated to multiple languages, and played by players all over the world, then, I suppose, there is no big RPG system from Japan.

However, there was, indeed, a table-talk RPG boom in Japan in the late 1980s, early 1990s.

A large role in this was played by Mizuno Ryō, who, at the time, was involved in translating and releasing D&D (the pre-Advanced rules) to the Japanese market. As a marketing stunt, they decided to serialize a transcript of a D&D session in the gaming and entertainment magazine Comptiq in 1986, to introduce the game to a larger audience. Mizuno was the GM and author of this campaign, which took place in his homebrew setting; this was serialized monthly between Sep. 1986 and July 1988.

This campaign ended up serving as the source material for the original Record of Lodoss War novels (published 1988-1993), which quickly moved away from the D&D ruleset and spawned an original—but heavily inspired by D&D—roleplaying system in 1989 (rev. ver. 1995). Lodoss became a huge hit, spawning several videogames, anime series, and novels (the most recent one was released two years ago).

Lodoss established Mizuno as one of Japan’s most well-known household names in fantasy. He subsequently moved on to serve as world designer for the Sword World RPG system, which ended up becoming the most commercially successful (and long-running) RPG system in the 1990s (and it's still ongoing). Sword World also spawned novels, videogames, and numerous “replays”: the original marketing move created a literary subgenre in the replay, a hybrid of light novel and a drama-scripting (up to this day, a big chunk of these still are Sword World-related publications).

Of course, there are also numerous other systems on the Japanese market, many of them showing a heavy influence of Akihabara (e.g., anime, manga, etc.) subculture; including translations of the major players from the West (D&D, World of Darkness, Call of Cthulhu, etc.).

Incidentally, the original Sword World setting, with its stronger “game-ish” world design than the more serious western fantasy (esp. the idea of a worldwide Adventurer’s Guild network became ubiquitous) has had a profound influence on the recent boom of webnovel fiction since the late 2000s: the settings in these stories (keyword: isekai fantasy) often are inspired by Sword World tropes (if they're not borrowing from MMORPGs instead). Goblin Slayer might be the worst offender: its setting is, almost 1:1, a very grimdark take on Sword World.

In sum, Sword World / Lodoss is the obvious candidate to being a “big” RPG, with numerous rulebooks, setting books, novels (etc.) that, in total, sold over ten million copies and that survived up to this day, but it has never been translated. And there might be no commercial incentive: we already had D&D and dozens of other settings on the market, which all are much more built towards the “serious” fantasy fiction (Lord of the Rings etc.) and to the stronger medieval-fantasy vibes people are used to in the West. (The idea of giving people a somewhat “watered-down” version of what they already have simply doesn’t sound like a good business move to me.)

But, of course, since no one ever bothered researching the Japanese TTPRG market and, in the process, interview both creators and business people on both sides of the continent on ideas like localization, your question can’t be answered beyond such a, cautious, speculative statement. Nevertheless, there are big Japanese TTRPG systems. In Japan.