Thankfully, this question has a short, simple answer! Prior to the ratification of the 25th Amendment in 1967, there was no procedure for Vice Presidential succession. Article II, Section 1, Clause 6 of the Constitution says that upon the President's death, removal from office, or incapacity, the Vice President will assume the role of President, but it doesn't say anything about replacing the Vice President. Prior to the ratification of the 25th Amendment, when the Vice President succeeded the President or left office in some other way, he wasn't replaced; instead, the position of Vice President was left open until the next Presidential election and filled by the winning Vice Presidential candidate. So, in Truman's case, he operated without a Vice President from Roosevelt's death in April 1945 until the January 1949 inauguration of Alben W. Barkley, his running mate in his successful 1948 re-election campaign. By the time the 25th Amendment was passed, the office of Vice President had been vacant for a total of almost 40 years.

Section 2 of the 25th Amendment provides that when the office of the Vice President becomes vacant, the President will nominate a new Vice President who must be confirmed by a majority of both houses of Congress. Thus, the situation of a Vice President succeeding the President or leaving office without a replacement and leaving a vacancy in the office of Vice President can't happen anymore. This issue has only come up twice since the ratification of the 25th Amendment: Spiro Agnew resigned in 1973 and Richard Nixon replaced him with Gerald Ford, and, after Nixon resigned in 1974, Ford succeeded him and replaced himself with Nelson Rockefeller.

As far as the rest of your question, if Truman himself had died before a new Vice President was installed, the answer of who would become his successor would actually have varied depending on when during his term it had happened; if it had happened prior to 1947, he would've been succeeded by the Secretary of State (James Byrnes or George Marshall), but if it had happened after 1947, it would've been the Speaker of the House (Joseph Martin). The Presidential line of succession is determined by the Presidential Succession Act, which Congress has the power to determine under Article II, Section 1, Clause 6. The original act, in 1792, placed the President Pro Tempore of the Senate next in line, followed by the Speaker of the House. In 1886, this was changed, and the Secretary of State was made the next in line to the Presidency. Truman didn't like this idea, because he thought the President should be an elected official, not a political appointee like a Cabinet member, so he pushed for a new Presidential Succession Act, which was passed in 1947. Ironically, had he died in office prior to his re-election in 1948, it would've resulted in his party losing control of the Presidency, since the Democrats lost control of the House in 1946. Fortunately for Truman, this didn't happen, and Barkley took over as his VP in January 1949. Thanks to the 25th Amendment, it's unlikely that the subsequent stages of the line of succession will ever need to be invoked, since the Vice President would be replaced upon leaving office, preventing a situation where both executive offices were simultaneously vacant.

Sources: Text of Article II, Section 1, Clause 6 of the Constitution; Text of the 25th Amendment; Text of the Presidential Succession Act of 1947