[META] AskHistorians answer quality prediction?

by qbxk

This is just a passing thought, but imagine correlating heuristics of question titles in this subreddit with some measure of the "quality" of the answers it has received in the past. This may allow for a useful "answer quality predictor" when someone is crafting their AskHistorians questions.

In effect, it would basically be giving you a measure of how much the community probably knows about that question, as you're phrasing it.

I'm fuzzier on how to numerically analyze the titles being submitted, but rating the quality of answers seems somewhat straightforward, and that the unique aspects of this subreddit lend themselves to that type of analysis. Some of these seem to me to be

  • the strictly moderated nature of it (producing many high quality posts to data mine from),
  • the somewhat limited vocabulary and frequently reoccurring keywords that are used in discussion (due to the detailed discussion of a relatively finite knowledge base)
  • the fact that karma tends to accumulate heavily in what are almost universally quality answers
  • the absence of any other kind of posts than high-quality ones, especially as a top-level comments

Just a thought over coffee this morning.

edit: formatting

Georgy_K_Zhukov

the fact that karma tends to accumulate heavily in what are almost universally quality answers

I would disagree, heavily, in fact. Karma tends to accumulate on answers which are to questions which are popular, and because of how reddit works, questions are generally upvoted well before there is an answer present, so the correlation is going to be small, at best. An answer which is visible is one which sufficiently meets the rules of the subreddit, but beyond that, there is huge variation.

To use myself as an example (and sorry to toot my own horn, but I know me best!), my top response to a question in this subreddit is this answer, right now sitting at 7,480 karma. It's OK, and I'm happy people read it, but it isn't anything special. It simply benefits from the fact that it is currently the 10th most popular thread of all time on the subreddit. If you wanted to know what my best work on the subreddit it, I would likely point you to either this one with 172 karma or this one with 125 karma, both of which would have been considerably lower than even that if they hadn't gotten /r/DepthHub X-posts. The questions just didn't get upvoted unfortunately, so the answers themselves flew mostly under the radar.

Edit: I have one more example to point to actually. This answer on Switzerland I'm pretty happy with, and it is the second most upvoted thing I wrote on the subreddit, at 4194 points. The thread is 51 all time based on a quick eye-balling. But that was 95 percent repost. I had done a few minor edits, but when I originally posted it, almost identical, it got only has 23 upvotes. Same answer, the main difference being that the thread was popular.