It's difficult, if not impossible, to give a confident answer for who was the "poorest" president, primarily because it's very hard to estimate assets from people living centuries ago. However, what I can do is provide a few general trends, and list a few presidents who were significantly less wealthy than most.
All the figures I present in this post are approximations calculated by Kathryn Moore in her very thorough book The American President: A Complete History. All figures are adjusted to 2012 USD, and represent the presidents' peak net worths (whether before, during, or after taking office).
From George Washington to Zachary Taylor, all of the first 12 US presidents were quite wealthy (i.e. multimillionaries). At the time, many Americans did not own land, and even fewer owned slaves and profitable plantations. Because of this, income inequality was quite high, and there was a steep divide between the upper class (which included the presidents of the time) and everyone else. George Washington is considered to be the richest POTUS in history, having owned over 100 slaves and huge tobacco plantations. Moore estimates his net worth to have been over half a billion. Of the first 12 presidents, Zachary Taylor was worth the least - about $8 million 2012 USD. This was still, of course, extraordinarily more than the average American was worth, but it nonetheless represented the transition to an era where US presidents were worth significantly less.
The next 21 individuals to be POTUS (Millard Fillmore to Dwight Eisenhower) were generally less wealthy than their predecessors. Moore speculates that this is because as the population became better educated, more citizens began questioning the idea that extremely rich people could best represent their needs. It was sometimes thought that the very rich, despite not having the same struggles as the layman, could best represent their needs because of their superior education and resources. However, the questioning of this assumption started becoming more common at the time. I could not find any other sources that back that claim up, so it's possible that this was mere speculation from Moore. Of these 21 presidents, 11 were worth less than a million dollars - a clear departure from the earlier trend. However, 4 presidents (Cleveland, Hoover, and the two Roosevelts) were each worth more than 30 mill 2012 USD, so the trend was by no means absolute. In response to your original question, the 11 presidents worth less than a million were Buchanan, Lincoln, Johnson, Grant, Garfield, Arthur, McKinley, Wilson, Harding, Coolidge, and Truman. It's impossible to say with certainty who among these was the poorest, because the records are simply not that detailed.
Starting with Kennedy, the past 10 presidents have been relatively wealthy, though not to the same degree as the first 12. They have all been multimillionaires, ranging from Obama (about 7 million) to Johnson (about 114 million).
To reiterate my source: The American President: A Complete History - Kathryn Moore
Comparisons are rendered difficult by variations in the value of money, but - as was noted in several of the deleted comments on this thread - there seems to be little doubt that Harry S. Truman was the poorest president since at least 1900.
Truman's relative poverty had several causes. He came from a farming family which was only moderately well off; he had to share the wealth of the farm with his siblings; and he made numerous bad investment decisions, including buying into a worthless zinc mine and setting up a failed men's haberdashery.
Among the more praiseworthy reasons for his lack of wealth were his determination to repay the $35,000 owed to his creditors in the men's store, rather than declaring bankruptcy as his business partner did, something that took him from the early 1920s to the mid-1930s to accomplish; his unwillingness, though part of the infamous Pendergast political machine in Missouri, to take bribes and dubious "commissions" on work allocated; and his absolute refusal, after leaving office, to take high paid directorships or lobbying fees or gifts (he refused, for example, the offer of a new Toyota, saying he would drive only American made cars) which, in his opinion, would have meant his trading on the fact that he had held the office of president, and hence cheapened the presidency itself. ( "I could never lend myself to any transaction, however respectable, that would commercialize on the prestige and dignity of the office of the presidency," he wrote.)
As a result, when Truman left office in 1952, he was so poorly off that he had no option but to return to his hometown of Independence, MO, and live in his wife's family home there. For a while he lived on his pension as a World War I army captain - $112.56 a month – and such savings (believed to be modest) as he and his wife had managed to put aside from his $75,000 a year presidential salary, which was taxed at a rate of just under 60%, during his second term. Truman's biographer David McCullough adds that, on leaving the White House, "Truman had been forced to take out a loan at the National Bank of Washington in his last weeks as President, to tide him over, though the amount was never disclosed." Bess Truman, meanwhile – who came from a supposedly significantly better off family, one that had seen Truman as a poor match for her – discovered on the death of her mother in the final days of her husband's presidency that her inheritance came to a meagre $8,385.90.
According to Truman's tax return for 1954 (available via the Truman Presidential Library), which seems to have seen the nadir of his post-presidential fortunes, his total income in that year was only $13,564.74.
Matters improved for Truman thanks to three circumstances. First, he negotiated to sell his memoirs to Life for $600,000 – a huge sum that was, nevertheless, paid in instalments between 1952 and 1960, and which was significantly reduced by tax and by the need to pay large costs to research assistants. Truman was especially angered to discover that he was punitively taxed on his book earnings, on the grounds that he was now a professional author; in contrast, Eisenhower paid only a non-professional's 25% capital gains tax on his war memoirs. Overall, his net income from the memoirs has been calculated as only $37,000. Second, Truman and his siblings sold part of the old family farm - 244 acres of good land that seems to have brought in somewhere north of $100,000 for Truman himself. Finally, once it became clear to Congress how financially embarrassed the former president was, Speaker Sam Rayburn and Senate Majority Leader LBJ passed a 1958 law that provided all former presidents with a $25,000 annual pension, plus additional funds to cover the running of an office, and free postage. The wealthy Hoover - the only other living former president at that point - graciously accepted the pension, though he had no need of it, in order to spare Truman further embarrassment.
McCullough makes this comparison between Truman and his predecessors:
Herbert Hoover, the country's other surviving former President, was a man of wealth who lived in style at the Waldorf Towers in New York. Theodore Roosevelt, William Howard Taft, Woodrow Wilson, none of those who had faced the prospect of being a former President since the start of the century, had worries about money; none but Calvin Coolidge, who, downcast over the death of his son and in poor health, lived less than three yeas after leaving office. Theodore Roosevelt had gone storming off to Africa to hunt big game; Taft became professor of law at Yale, then Chief Justice of the United States. Wilson, with his wealthy second wife, retired in style to a twenty-two room house on S Street in Washington..."
[Truman pp.928-9]
With regard to your question as to this wealth (or lack of it) in present day terms: Truman was a senator when he was selected as Vice Presidential pick for the 1944 election – which, given that he automatically succeeded FDR on the latter's death not long into his fourth term, seems the fairest point to take when calculating his "wealth when elected." His salary at that point was $10,000 a year, on which he paid just under 29% tax – and from what we know of his finances, he was essentially entirely dependent on that to support himself and his wife and daughter in Washington, which was a significantly more expensive place to live than was Missouri. The US Bureau of Labor Historical Inflation Calculator indicates that $10,000 in 1944 was equivalent to $137,132.39 in 2016 – hence the Truman family was living on a net income of around $97,000 in current terms when Truman was selected to join the Democrat ticket. As Vice President, by the way, his salary was increased to $15,000.
McCullough's biography is full of asides concerning the difficulties the Trumans had making ends meet on this salary, and the economies that Bess and Harry felt forced to implement, even after Truman became president. (Although the presidency naturally carried a far higher salary, as indicated above, occupants of the White House were expected to pay for a lot of official expenditure, such as entertainment, out of that sum. Previous occupants had funded quite extravagant entertainment of official guests out of their personal resources; the Trumans were unable to do this.)
Are you wanting to take in land value and things like stock?
May I ask a specific follow up question? Where does Nixon stand in all of this? I know he started life relatively poor but where did he end up post presidency (financially speaking, I know he ended up in CA)?