I hope someone can help with the specifics to answer your questions about colonial America and differences between Northern and Southern practices. The traditional Halloween as it is known today - like so many things - took its final form after WWII to cater to the emerging and massive Baby Boom generation, but it has ancient roots and was a tradition that meandered through time and space, until its transformation was complete. The following is how I have treated the subject in my Introduction to Folklore. I hope this helps - I'm open to questions. Here is he excerpt:

Of all the holidays of the American calendar, Halloween has perhaps the most ancient roots. Its origins go back to a pagan autumn celebration popular in some parts of Western Europe long before the Christian era. Many prehistoric Europeans regarded November 1 as a time to remember the dead. Finding it impossible to discourage this innocent devotion, the early popes decided to give a Christian veneer to the pagan holiday. The feast of the dead became a mass for all the saints, known in an earlier form of English as “All Hallows’ Mass.”

The night before was “All Hallows’ Eve,” a name shortened to Halloween. Priests successfully directed the folk to Christian endeavors during the day of November 1, but the preceding night remained the domain of the wild pagan imagination. People believed that on Halloween, when the sun set, the gate separating this world from the next opened and the dead walked among the living.

The withering vegetation of autumn must have impressed ancient Europeans. Dependent on what food the land had to offer, they saw this as a terrifying event. The death of the world would mean their starvation, and so this became a time to petition the dead for help. People believed that their ancestors could sway the forces of nature to ensure the return of life the next spring.

This is reminiscent of the Christian practice of praying to the saints on November 1 to encourage them to intercede with God on behalf of the living. Long before conversion to Christianity, many Europeans had ritual celebrations intended to influence the dead. For the Romans, it was November 8, and for the Celts, it was November 1. The Irish still retain the ancient term, Samhain (pronounced sa-win) for this holiday.

With or without the Christian veneer, this was a time when anything could happen. People believed that witches gathered to make special magic that could only be brewed during the death festival. It was also the night when all sorts of supernatural beings besides the dead were afoot. Demons and imps traveled the land. In Britain, one of the most famous of these was Will-’o-the-Wisp, also known as Jack-’o-Lantern. This entity would lure people into swamps with a dimly-lit torch mimicking the distant light of a farmhouse.

Many different legends account for Will-’o-the-Wisp. Some said that it was the spirit of a man who could not rest because of a wrong committed during life. Others said that it was the spirit of a clever but evil man named Will or Jack. After death, Will’s soul went to heaven where St. Peter recounted the evil deeds of his life. St. Peter refused him entry into heaven and ordered him to hell. Will walked to hell, where he tricked the devil into a bag and beat him or in some other way made the devil fear him. The devil consequently ordered him out of hell. Will’s exile condemned him to walk the earth, unwelcome in both heaven and hell. Before he left hell, however, Will stole a piece of burning coal, thinking it would keep him warm. This is the flickering ember that distracts lost travelers, inspiring the carved glowing pumpkin – it was originally a carved turnip in Britain – with a hideous face.

The idea of the Jack-’o-Lantern comes from English and Irish tradition. In fact, it is from the Celts that America inherits most of its notions of what Halloween is supposed to be. Today, the Irish continue the prehistoric traditions of lighting bonfires on Halloween, a practice that many American campuses have translated into homecoming week in autumn. In England, this serves as a root of the November 5 Guy Fawkes celebration, with its bonfires.

The American practice of trick-or-treat derives from the belief that Halloween was a time when spirits walked the earth, but there were other motifs that played into this custom. In the Celtic countries of Western Europe, it was once common to make sacrifices to the dead, leaving food at graves. Some even set the table before retiring for the night so that the visiting dead could enjoy a midnight feast. In turn, the dead were said to leave presents for children. In Christmas-like fashion, youngsters found gifts the following morning.

It was also customary for people to make candy for the dead. Eventually, bakers created elaborate sugar statues depicting Death marching its macabre harvest of corpses and skeletons to the grave. This practice has reached its most elaborate development in Mexico with the celebration of the Day of the Dead.