This is covered in the thread on the SR-71 in more detail, but the essential point is that both of those Soviet fighters are really interceptors through and through- their only mission is to protect local airspace from bombers and recon planes, so they can afford to sort of min-max for climb rate and service ceiling, without needing to prioritize range, maneuverability, or the ability to perform a range of missions. The Tomcat and the Viper were originally conceived as a multi-role naval fighter and air superiority fighter, respectively. This meant that they needed things like loiter time, range, maneuverability, long-range radar, a variety of weapon options, and the like. Building these features into a plane that also had exceptional service ceiling would be more difficult and add to costs.

Because the 25 and subsequent 31 (a redesigned and updated 25) were built for the specific purpose of shooting down American (nuclear) bombers attacking the Soviet Union.

Think about that scenario for a moment. You are the USSR and you have to defend a massive landmass against attacking bombers and potential spy planes, during a time when military aviation more and more went the route of 'higher and faster'.

The Soviets knew about the high flying U-2 and the Americans were working on bomber concepts that would fly very high and very fast. Supersonic at least and there were also ideas for Mach 2+ capable bombers like the (heinously expensive and ultimately cancelled) Valkyrie project (XB-70). In addition to the supersonic+ B-58s already in service.

So, you have to defend your huge landmass versus enemy aircraft that presumably fly very high and very fast.

So you build a big bird that can fly very high and very fast, with decent range. Because that's how you catch those enemies.

The MiG-25 is not really directly comparable with the Tomcat, Falcon or Eagle like that. The 25 was built with a very specific role in mind, it's not like your standard air superiority fighter.

The MiG-31 isn't really comparable to any American plane. It was designed specifically as a high-speed/high-altitude bomber interceptor. It has very unique features, a unique mission and is not really designed to go against other fighters (it's not maneuverable at intercept). Therefore, I think it should be excluded from any performance comparison with American aircraft.

The MiG-25 and F-15 are better-suited for comparison. Both entered service in the early-1970s and were designed primarily as air superiority aircraft. The MiG-25 has a service ceiling of 67,900 ft and the F-15 has a service ceiling of 65,000 ft. While the Foxbat slightly outperforms the Eagle in ceiling, it greatly lacks in other essential fighter performance characteristics such as rate of climb (> 50,000 fpm vs. 41,000 fpm), range (1,200 NM vs 935 nm) and maximum g load (9 g vs. 4.5 g).

All in all, I don't have a perfect answer for you. Military design is one of trade-offs. You sacrifice in certain areas to be dominant in other areas. The Foxbat may have a slightly better ceiling but it pales in comparison to the Eagle in other performance metrics. I have no knowledge of whether this was intentional by the MiG's designers or merely a by-product of their design.

Soviet design philosophy was more focused on a specific roll for each aircraft. The rolls of the aircraft you mentioned are not the same, either. The MiG-25 evolved into the MiG-31... both were dedicated interceptors. The MiG-29 and Su-27 were designed to be land and sea based fighters respectively. The F-14 was a huge generalist... it had a ton of rolls from air superiority fighter, to reconnaissance; it was an interceptor but the Soviets didn't field anything like what the US had so the give and take on design was different. I've seen anti-satellite missiles mounted on F-14s, not to mention the huge Phoenix missile which was meant to take out Tu-95 Bear bombers. The F-16 got more rolls later, but if there's a pure "air-to-air" fighter in the US inventory it's got to be the F-16. It's amazing to this day. As interceptors, the MiG-25/31 needed to catch up to their prey, which included the SR-71. Thus, they needed to be capable of reaching high speeds and have a very high service ceiling. There wasn't a Soviet equivalent to the SR-71. We just didn't need to go that high. US air-to-air missiles were sufficient to reach any soviet aircraft. The soviets ended up more than catching up to the US on that front though.

The more I think about this question the more wordy the answer gets. The MiG-25 was not really a projection system. It's didn't last long under full burn,, but it didn't need to. It was built to defend Soviet airspace from US overflights. It could be assumed there would be a friendly runway nearby, so it made sense to sacrifice endurance for speed and service ceiling. The F-14, conversely, WAS a projection system. It's variable wings, when swept forward, could greatly increase the range. There are a lot of considerations that go into each aircraft's design. It's a big game of give and take. I'm not sure the answer to your question is in what I've written, but I hope it helps.