They couldn't have. Until the passage of the 25th Amendment in 1967, there was no way to fill a vacancy in the office of Vice-President before the next election.

Before then, there had been 15 occasions on which the office of Vice-President had been vacant since the first Vice-President took office. Many were significantly longer than Johnson's first term, including the terms of Andrew Johnson, Chester A. Arthur and Theodore Roosevelt (first term)- all of whom had been VP, succeeded to the Presidency after the President was assassinated, and served well over 3 years without a Vice-President.

There were also several elected Presidents who served much of their term with no VP after the VP died. Franklin Pierce's VP, William R. King, died in office after only 45 days.

Had something happened to a President who had no VP, the President pro tempore of the Senate (before 1886), Secretary of State (between 1886 and 1947) or Speaker of the House (after 1947) would have succeeded him. This almost happened with Andrew Johnson when he was impeached and the Senate was one vote from convicting him.

At the time of President Kennedy's assassination, there wasn't a way to appoint a new vice-president except through a regular election. In 1965 Congress proposed the 25th Amendment, which specified the method to fill a vacancy in the office of Vice President. It was ratified by enough states on February 10, 1967 when both Minnesota and Nevada approved it. There have been two replacement VPs after this: Gerald Ford replaced Spiro Agnew when Agnew resigned before he pleaded no contest to bribery. Nelson Rockefeller replaced Gerald Ford when Ford became President after Nixon's resignation.

There is a section relating to succession in Article Ii, Section 1, Clause 6, but it doesn't mention a need to replace a Vice President upon the death of a President. In fact, when William Henry Harrison became the first President to die in office, his VP John Tyler's assumption of the full office of the President was controversial and considered by some to be illegal. These people believed that the Constitution only allowed the Vice President to assume the duties in an "acting" capacity. The 25th Amendment finally codified the succession process as Tyler acted.

Finally, the 25th Amendment included specific processes for temporary removal of a President due to incapacity. This has been invoked when a President was undergoing surgery or procedure that made him unable to carry out his duties. One of the other purposes of this was to avoid a situation such as when Woodrow Wilson had a stroke and was unable to perform his duties, while his wife acted as "President". The Amendment now allows the cabinet and Vice President to declare the President unable to perform his duties. This part of the Amendment has never been invoked.