To answer the first part of your question:

The Constitution did not specify March 4, or any other date, as the beginning of the new President's term. It stated only that the length of the President's term was four years. And, while the Constitution gave Congress the power to set the date for the selection of Electors, and the date the Electors should cast their votes, it did not give Congress the power to set the date on which the President's term began.

Congress met for the first time on March 4th 1789. This date was the result of a vote of the Continental Congress:

Resolved, That the first Wednesday in January next be the day for appointing electors in the several States, which before the said day shall have ratified the said Constitution; that the first Wednesday in February next be the day for the electors to assemble in their respective States and vote for a President; and that the first Wednesday in March next be the time, and the present seat of Congress [New York] the place for commencing, proceeding under the said Constitution."

[EDIT: So, to answer OP's specific question, it looks as if they were fond of Wednesdays, and March 4th happened to be a Wednesday in 1789].

Although Congress didn't count the electoral votes of the first Presidential election until April 6th of that year, and President Washington was not sworn in until April 30th, his term in office was retroactively considered to have begun on March 4th (since that was when the new Constitution "commenced").

Since every Presidential term was four years long, as specified by the Constituion, every subsequent Presidential term (until the XX Amendment was enacted) also began on March 4th.

An interesting side note is that March 4, 1848 fell on a Sunday. Zachary Taylor refused to be sworn in on a Sunday, and took his oath on Monday. Millard Fillmore, Taylor's Vice President, also waited. Polk did not stay in office for that extra day. Some claim David Rice Atchison, president pro tempore and next in succession, was president for that day. He never claimed to have been president for that day (in fact mentioned he slept for much of that day), and the myth likely began as a marketing ploy by Atchison, Kansas.

January 20th is roughly a month after the Elecorial College vote (the third Wednesday in December) and roughly two weeks after the president of the senate confirms the vote. January 20, 1937 was a Wednesday, so the 20th became the official day (FDR was already in office and the 20th Amendment reached the required number of states ratifying it on January 23, so FDR's second term retroactively began on the 20th). To keep terms four years, it has remained January 20th.

ETA: I stand corrected. When arguing to shrink the lame duck period, those explaining why a long lame duck can be a bad thing, point to secession of Southern states with the election of Lincoln and the bank failures in the Great Depression (which FDR addressed pretty much as soon as he was sworn-in in 1932). For some reason I associated the 20th Amendment as a reaction to FDR closing the banks. Instead it was more of a reflection of larger issues related to the ability of government to respond to crisises.