This brochure from Massachusetts Department of Conservation and Recreation says:

In September of 1847, Thoreau completed his experi- ment in simplicity and became a sojourner in civilized life again. Emerson then sold the house to his gardener. Two years later two farmers bought it and moved it to the other side of Concord where they used it to store grain. In 1868, they dismantled it for scrap lumber and put the roof on an outbuilding.

Walden, the book, wasn’t published until 1854, seven years after his two year stay by the Pond (which he condenses into one year for the book) and years after the house was sold and moved. Remember that the Emerson family actually owned the area.

Thoreau is buried in the main cemetery in Concord, MA, then I believe attached to the First Parish Church and called the New Burial Ground or officially Sleepy Hallow Cemetery (it was established in 1855, only a few years before Thoreau’s death, a little further from the Milldam and the Town Green than the older cemeteries; Emerson spoke at its dedication, as did William Ellery Channing a Unitarian theologian who was very well known at the time). It’s still an active cemetery, run by the town; here’s their official website. At some point, probably 1866, Thoreau’s grave was moved to a spot know as “Authors’ Ridge”, where the luminaries of the Concord literary scene were buried. People often leave gifts and trinkets as votive offerings on these graves; it seems particularly common to leave coins, small stones, flowers, pinecones, letters, and especially pens and pencils by these graves (Thoreau’s, Louisa May Alcott’s, Hawthorne’s, Emerson’s). Pennies are common on many of the graves because Daniel French, the designer of the Lincoln Memorial (the reverse of the penny from 1959-2008), is also buried there. Pencils are somewhat more common on Thoreau’s grave than pens because Thoreau’s family owned a pencil factory. I have no idea when these traditions started but long before I was born, at least (I grew up in a neighboring town and learned to kayak on Walden Pond before I ever read Walden). It seems likely that he as moved to Authors’ Ridge within a few years of his death because people were already visiting.

You’re right, though, to expect pilgrims to come and want to see “where the philosopher guy lived”. Visitors eventually started setting up a cairn (a ceremonial pile of rocks). This is the 1872 event you refer to, and at this point Thoreau was already ten years dead, and the book had been out for slightly less than 20. Bronson Alcott (Louisa May Alcott’s father) had a guest visiting from Iowa, Mary Newbury Adams, who left the first stone and Alcott himself left the second. Alcott wrote in his journal entry of July 12-13, 1872,

Mrs. Adams suggests that visitors to Walden shall bring a stone for Thoreauf’s monument and begins the pile by laying down stones on the site of his hermitage, which I point out to her. The tribute thus rendered to our friend may, as the years pass, become a pile to his memory. The rude stones were a monument more fitting than the costliest carving of the artist. Henry's fame is sure to brighten with years, and this spot be visited by admiring readers of his works.

Since then, many other famous and not famous writers have come and left stones, with members of a literary magazine visiting in 1878 (possibly capturing the first picture of the cairn) and Walt Whitman visiting in 1881. Around the centennial of Thoreau’s sojourn by the Pond, there was a minor dispute among Thoreau enthusiasts about where exactly the cabin was, whether it wasn’t a little further from the shore or even at an entirely different cove. An amateur archeologist did some excavations and found the base of the chimney of Thoreau’s cabin near the cairn, meaning that Alcott remembered more or less the correct spot. The back of the cairn covered about 1/4 of the cabin site, and stones from the back of the cairn were moved to front so as to clear the site. Shortly after the site’s excavation, granite posts (with granite from Thoreau’s birth place) marked the edges of where the house stood, though many tourist mistake the replica (close to the visitors’ center, on the other side of the road) for the original. The cairn was never really made an official monument, and in 1975, the Commonwealth’s Department of Environmental Management, which was then responsible for the Pond, decided to haul away the massive pile of rocks that the cairn had become, without much warning. After much public outcry, the offending party returned “the same rocks” to recreate the cairn, three years later. When it was returned, it was moved slightly further away from the house (since it was no longer needed to mark the actual site of the house, which had its own markers since the 1940’s). The best history of the cairn is here from a Thoreau Society newsletter, which also includes information about his grave site.

So, while the cabin had already been sold and moved years before the book even came out, shortly after Thoreau’s death, memorialization of him began at two sites, the Pond and his grave.