There are generally three requirements for the hajj:

• Mental and intellectual maturity, so one can understand the significance of the pilgrimage

• proper physical health to facilitate the pilgrimage

• proper financial conditions to facilitate the pilgrimage.

So very simply, the Hajj though mandatory is only for those capable of making the journey. Those too poor to do so, than as now, are exempt.

Edit: for some textual grounding to this claim 3:27

In it are clear signs [such as] the standing place of Abraham. And whoever enters it shall be safe. And [due] to Allah from the people is a pilgrimage to the House - for whoever is able to find thereto a way. But whoever disbelieves - then indeed, Allah is free from need of the worlds.

Yes.....and no. In general, it's much easier to go for hajj now than it was in the past. There's a very famous travel log written by a man name Ibn Jubayr which chronicles his experience traveling from Andalus to Mecca. The catalyst for that journey was that he had received a large sum of gold from a prince who felt bad for forcing him to drink wine. The point being, if he hadn't received that gold, he would not have gone for hajj. This is, however, for those living far way. If you were a poor farmer in southern Arabia, it wouldn't be too difficult to make arrangements so that you could go for hajj and come back in a relatively short period of time.

Nowadays, a higher end hajj is still probably less than a fraction of an average person's yearly income. In the past though, hajj was a massive financial undertaking if you lived far from Makkah. You would need money for transport (camels, ship fees, food on the way, etc), you would need money for bribes (plenty of enterprising bandits exacted a toll for pilgrims who passed though), and you would need to have enough money stored away to support your family while you were gone. Basically, for people living far way from Makkah, it was very unlikely that they would have gone for hajj. We know this because people who did go for hajj were even given the title hajji. In other words, going for hajj was such a big deal that people would refer to you as "Hajji So and So" after you returned back.

Edit: I probably should have clarified. Hajj is not obligatory upon those who do not have the money or physical capability to do so. So a poor person wouldn't be missing an obligation (although I'm sure they would still have preferred to go if they could).