See his signature in his wiki page for an example. In general he seems to sometimes replace a terminal s with a z but not always. See first result for james k polk handwriting. I can't find any source that explains the behavior as some sort of normal cursive standard of the time.
As you note, he often uses this convention in general writing. For example, see the 's' at the end of 'States' on p5 of this letter from the Polk papers. As to why he followed that convention, it's almost certainly something he learned from his father, Samuel Polk (surveyor and soldier), who did the same thing. See his letter to Andrew Jackson from 1815, specifically the word "Surplus" at the end of the very first paragraph. Inconsistencies in the use of the terminal 's' with tail in the same letter may be oversights, like the many on other pages in those same letters, but it seems like most if not all of the tailed letters appear on capitalized (not always proper) nouns (James, States, Surplus), which is interesting.
As to why Samuel Polk did that, well, I'm not exactly sure--we don't know that much about his education, except that it was sufficient for him to become an officer and a surveyor. It wasn't his own father, Ezekiel; we have one letter from Ezekiel Polk to Jackson from 1813, and the terminal 's' is not tailed in that way. This assumes, of course, that Ezekiel wrote his own letters at age 65, but we can't know unless we find some other indication. Many of Ezekiel's own papers are at the Tennessee State Library and Archives. There is a handlist available here but I did not find any sign of digital collections. The papers have been microfilmed, but again, I don't see any scans of those out there. (ed: Any writing by Mary Wilson Polk, which might also offer a source, would be there I reckon.)
So to answer your immediate question, it was a rendering of the terminal 's' that he learned the habit from his father, and it was possibly reinforced in other ways or at least not discouraged (at the Bradley Academy, perhaps); as to his father's source for the practice, we don't know for sure. Someone else may be able to chime in with information about where such a practice might have arisen and then been taught in the late 18th century. In my own archival work, I have never encountered the practice, but then I tend to stay on the other side of the Atlantic.
(ed: added link for finding aid)