After he was elected, Washington moved to the then-national capital of New York City, where Congress had rented a residence for him at 3 Cherry St. The house was somewhat small for official business, and so he moved to 39-41 Broadway. When the national capital moved to Philadelphia in 1790, he took up residence in the Robert Morris mansion on High Street; he and John Adams both lived there, with Adams moving into the White House once it was finished in 1800.

As to "calls to abandon Mt. Vernon," it's been awhile since I directly studied the vey early Republic but I can't imagine they would have been necessary; in the days before electronic messaging you had to be where the work was, and it was clear that his position was where the government sat. Unless you mean something different by "abandon."