We discussed this quite a bit in my 18th Century British Novel class. You're on the right track with advertising. The long titles got people interested and piqued their curiosity. Also, keep in mind that the 18th century marked the very beginnings of the novel itself. Up until then, there were none. There were epics and romances, but not novels as we know them now. Many of those in the upper class viewed them as not worthy of their time, and many authors of the first novel were writing for the growing middle class, which was becoming literate in large numbers for the first time.

Someone else will probably have a better answer. I'd love to read it.

As far as the spelling change you mentioned, I'd say that could just be attributed to language evolution. That was pretty common (and still is). But someone at /r/linguistics would definitely be able to answer that one more specifically. It likely has to do with vowel shifts, but I'm not a linguist.

There isn't a really good answer as to why titles were so long. They could just as easily ask us why our titles now are so short. That being said, there are a few considerations that could help us to understand why they did it so differently than us. I'm just building on what dark_haired_girl pointed out so well, that the novel had to find ways to promote itself in a time before it asserted its cultural dominance in the 19th century.

Also, your question should really be about pre-modern (generally 17th century) and 18th century books, because the 19th century had generally ended the practice of these long-winded titles. As I pointed out here the serialized novel in the 19th century could not afford to have such long titles, since space in the newspaper or magazine was very precious. The 19th century was also the beginning of standardized printing practices that allowed for a "trade binding": a book cover with art and promotional material. I always use the beautiful example of the Hetzel editions of Jules Verne to show how very commercial the book cover business had become by the mid 19th century. The 18th century hadn't yet adopted these advertising practices, so the interest of books had to be explained in plain language on the first page.

People at the time fully understood that the long titles were basically secondary description around the title itself. No one referred to Robinson Crusoe as "The Life and Strange Surprizing Adventures of Robinson Crusoe, Of York, Mariner etc. etc". Even when the title was something short(ish) like "Candide, ou l'optimisme" ("Candide, or Optimism") from 1759, no one would have thought to call Voltaire's book "Optimism," as they all knew the real title was "Candide."

To give you a better idea of how these titles functioned, take a closer look at Candide. Even though its title isn't as rambling as others in the 18th century, its chapter divisions truly are. The first beings with "How Candide was raised in a beautiful castle, and how he was chased from it." This isn't really a chapter "title," it's more of a table of contents. The book is announcing to you what you'll be encountering in this part of the book. You might also recognize this practice in Don Quixote (or any number of other pre-19th century books), which begins each of its chapters with a run-down of its contents. Now, we're back in familiar territory. Contemporary readers would not at all find it strange that a book would provide you with a table of categorical divisions, with some sort of indication of what each division contains. In fact, many publishers require books to have a table of contents. The practice is less-prevalent with novels, but it's still not unheard of.

The long titles served the same purpose as these chapter introductions: they prepared the reader to encounter the book. They gave the reader a good idea of what to expect, and in doing so, presented itself as a desirable thing to read. You say that this spoils the plot, but they usually tried to avoid giving specifics away. In your example of Robinson Crusoe, that last line - "strangely delivr'd by Pyrates" - actually doesn't explain anything. It poses more questions that it answers. Why did the pirates free him? How did he negotiate with them? How did he manage to stay alive?

Robert Darnton wrote a lot about the history of the book in the 18th century. His book "The Business of the Enlightenment: a Publishing History of the Encyclopedie 1775-1800" is a great start. Gerard Genette's "Seuils" is a good French-language theoretical framework that deals with a book's "thresholds": its title, the book's binding, the table of contents. If you read French, Genette's book is a real pleasure. Finally, if you ever need a reason to learn French, do it so you can read "Histoire de l'édition française" (dir: Roger Chartier), which is a four-volume series of essays on the history of the book in France. Never have I seen such a monumental book in a relatively new field.

You also have to remember that most novels started out serialized back then. Novels would be published one chapter at a time in magazines, journals, or even newspapers, so one novel might be serialized for a year or more in a magazine before it was actually finished. One reason for the extra-long titles would be to pique the attention of new readers and give them a synopsis of the plot, so they wouldn't be completely lost.

Well, with 12 Years a Slave, it was a true story that was in the papers. By providing all that information, people could be more certain that that was the story they'd read about in the paper. So it was advertising of a sort. And with Robinson Crusoe, that was somewhat based on a true story [of Alexander Selkirk]--a true story that had piqued the interest of the public. The long title may have served to remind people of that original tale. On the other hand, novels of that time didn't necessarily have such long titles. They needed to convey the propriety of the book quickly, though. Certain books would not have been considered proper for an unmarried woman, for example. Tom Jones was entirely fictional, published in 1749, and had the full title The History of Tom Jones, a Foundling. A pretty short title that nevertheless showed it might be risque. But each chapter had a really long title that often mocked the convention of long titles, as the chapter titles told in detail what was going to happen in the chapter, or else were titled things like "Containing LIttle or Nothing" or "Being the Shortest Chapter in this Book."

On a tangential note, early films did the same thing, with text placards that would delineate the scene yet to unfold.

small hanger question, why did "pyrates" change to "pirates" over time?

From what I've been able to find, 'pyrate' is simply a variant spelling of 'pirate' that still shows up today. According to etymonline, the spelling 'pirate' dates to around 1300, from the Latin piratus, replacing Old English sæsceaða. So 'pyrate' didn't change to 'pirate', it was just another way to spell it.