Not sure about access during the White House years, but in Chernow's bio of Washington and Gordon-Reade's book on Jefferson and Sally Hemmings, both ex-Presidents complained about the cost of unplanned visitors to Mount Vernon and Monticello. Apparently it was perfectly fine for a complete stranger to show up at an ex-President's home, totally unnanounced and uninvited, and expect the ex-POTUS to feed, lodge, and entertain them at no cost, as well as their horse(s). Often there would be several of these uninvited people at the same time. After years of service to the nation, Washington had looked forward to spending quality quiet time with Martha, and lamented the fact that night after night he was forced to make small talk with total strangers. Washington could absorb the costs of entertaining these strangers, but the situation greatly contributed to Jefferson being destitute at the end of his life.

I'm not a historian, but I recently read Destiny of the Republic: A Tale of Madness, Medicine and the Murder of a President by Candice Millard. In the book. Millard talks about how Garfield's assassin, Charles Guiteau, would visit the White House every day planning to sit and meet with Garfield to discuss a position within the cabinet. He got at least one face-to-face meeting. The White House also held receptions open to anyone that wished to attend. I'd have to sift through the entire book to find specific mention of what you asked for, but in any case visitors were screened by the President's private secretary Joseph Stanley Brown and either got to sit with the President or not.

It's pretty wild to consider but at that time, even though Lincoln had been assassinated only 16 years earlier, the general attitude was that because the American population had the power to choose the President there was little cause for angry rebellion. It was also general consensus that even if he was at a higher risk than the average person, nothing could be done about it anyway.

Edit: OK I found something. Garfield held calling hours of 10:30 a.m. to 1:30 p.m. and the White House typically had long lines of people seeking office or audience with him. It's probably mentioned in the book that Presidents were always very accessible, but I haven't found it yet.

Lincoln was well known to open the doors and allow lines of citizens and office seekers throughout the halls of the white house. This was in the local newspaper It is one of the tribulations which must greatly add to the fatigues of office at his juncture, that our amiable President has to give so much of his time and attention to persons who apparently having no business of their own, expend a large degree of their surplus energy in benevolently minding the business of the President.

And William Stoddard is quoted as saying It is hard work for the President, but he receives and dismisses these varied hundreds of people with wonderful dexterity, the trained result of his natural tact and wit and long practice.

In the prologue of "The Rise of Theodore Roosevelt" Edmund Morris relates the tale from the new year of 1907 when, as per his tradition, Roosevelt opened the White House to give a new years greeting to any sober and clean citizen who wished to come by. Describing the scene, Morris writes as follows: "[Roosevelt] stands with tiny feet spraddled, shoulders thrown back, chest and stomach crescent as a peacock, his left thumb comfortably hooked into a vest pocket. For what must be the three thousandth time, his right arm shoots out. 'Dee-LIGHTED!" Unlike his predecesors, Theodore Roosevelt does not limply allow himself to be shaken. He seizes on the fingers of every guest and wrings them with surprising power. "It's a very full and very firm grip," warns one newspaper, "that might bring a woman to her knees if she wore her rings on her right hand." The grip is accompanied by a discreet, but irresistible sideways pull, the the President, when he lets go, wishes to have his guest already well out of the way. Yet this lightening moment of contact is enough for him to transmit the full voltage of his charm."

So, at least in Theodore Roosevelt's time, while a citizen could not necessarily spontaneously visit the White House, it was opened for special occasions. And, given Roosevelt's gregarious nature, there would be a good chance of a short conversation with him. Ask him about his Roughriders and he would probably talk your ear off. The day which Edmund Morris is describing is actually additionally remarkable because Theodore Roosevelt set a long standing record for most hands shaken by one man in a day.

I love the story about the 1829 inaguration of Andrew Jackson. The open reception at the White House turned into a drunken brawl. The president had to escape through a window.

[Lincoln] saw the men ahead. There was no way to avoid them. The guards could not seem to keep them out, and many of them slept in the White House hall. The word had passed that he was coming, so they were on their feet and smiling. Each of these wanted a favor. As he passed, hardly pausing, they asked for jobs or passes to Richmond or the commutation of a military sentence or presidential approval of an illegal business deal. In four years of living in the White House, Mr. Lincoln had become accustomed to the morning vultures. He could do little to be rid of them, and had no desire to help them because, if their claims were just, they would have satisfaction at the proper agency.

This arresting passage is from The Day That Lincoln Was Shot, the popular history book by Jim Bishop. Describes what happened when the president stepped out into the hall from his White House bedroom on the morning he was assassinated.