Hello everyone, and welcome to Part One of "AskHistorians 2017 in Statistics". For the original Statistical Snapshot, 'taken' in April, check out this previous thread.

As always, I'll start with the short and sweet. Over the past year popular threads - referred to as the "T50", as I approximate this with the top 50 threads per month - averaged a 96 percent response rate, consistent with 2016 as well. The median actually increased from 96 to 98 percent from 2016 to 2017, and we even hit a "perfect" 100 percent twice. For the overall subreddit, using seven 24 hour 'snapshots', the response rate was 0.38, a drop from 0.43 percent in 2016, but mostly related to the average increase of 10 threads per day! In absolute numbers, the 2017 46.95 responses per day is not far from from the 47.61 per day in 2016.

Now a few notes!

First, people ask how I tally all this up. For the T50, import search results into an Excel sheet to run calculations off of. However, when calculating the response rates for the daily snapshots, that is done by hand! Have a screen open, I open all the threads up, and... use tally-marks on a sheet of paper to track the numbers. Those results then go into the spreadsheet, but given the nature of deducing "is this answered?" it just isn't easy to automate.

Beyond that, to rehash a little of what those who read the last thread will already know. There are two core statistics used when judging a thread, the "Response Rate" and the "Answer Rate". The first includes threads which receive a link to a relevant FAQ page, or a previous answer to the same question. As the important factor is engagement, if the link is by the original author, that is counted as an Answer, not a Response. As for what counts as an Answer, these is very little judgement. While there are a few threads I encountered time-to-time which clearly managed to go under the radar and something clearly rulesbreaking was remaining, which I don't count, on the whole as long as there is a visible answer to the question, it counts for the stats, whether barely sufficient or the best thing I read all year.

Finally, I'll offer a little analysis at with each statistic, and also at the end there are some further notes on the calculations/methodology.

The first group of statistics is a study of the Top Posts for a given month. This evaluates the likelihood of responses to the 50 most upvoted threads of a given month, which roughly approximates the threads most likely to have hit the top spot in the sub for that month, and thus be visible on /r/All, or /r/Frontpage. It also evaluates the time in which it took answers to arrive.

TABLE I: Monthly Top Thread Statistics - "T50"

So, to state the obvious, things have been pretty consistent for the past two years with the T50. There was a bit of a dip at the beginning of the year, but finished fairly strong! It shouldn't really surprise anyone that visibility can help ensure an answer to show up, but still, given how bizarre some questions can be, it does consistently impress me. The time it takes to see an answer show up has increased a bit, but as we like to say, with a little patience, one will likely show up. How, what questions didn't get any viable response? Well, although I'm a terrible statistician and only started saving the data part way through the year, I have half of the years links saved:

How did the tonfa, a weapon used in an okinawan martial art become a weapon used by police forces worldwide?

Every president from JFK to Reagan faced an assassination attempt (except for LBJ). Why were there so many attempted assassinations?

When/where did the stereotype of fat Americans eating McDonalds all day develop? In particular, did communist propaganda use this trope?

In 1967, the third season of Batman replaced a white actress with Eartha Kitt, an African-American woman. What was the response to this? Was it a controversial decision?

Christianity in the Roman empire went from its greatest period of persecution, under Diocletian, to legalized status, through the edict of Milan, in less than ten years. How did contemporary non-Christians react to this, and how foreseeable was it that Rome would become a Christian empire?

The last one is the odd-duck, being a field which does have decent coverage on the subreddit, but otherwise, the consistent theme is that they are questions which don't necessarily fit into the major flair coverages we have (Know anyone who does post-WWII American culture? Send 'em our way!).

The next two tables are based off of seven 24 hour snapshots per month, with the intention of taking the larger view of the subreddit. This comes to a total of 84 days evaluated, or 23 percent of the year's threads! It is presented in both the raw numbers and the percentages:

TABLE II: Monthly Snapshot by Percent

And the same in raw numbers:

TABLE III: Monthly Snapshot by Numbers

So, as you can see, things were pretty consistent through the year. Graphed out for a trendline, you get the Response numbers for 2017 as "y=-0.2517X+330.3", and if it weren't for unusual outlier at the beginning of 2016's data, it would hold true through 2016 as well, as "y=0.3269X+320.19" (Add them in, and we get "y=-1.3596X+347.95). But that is the raw numbers. If we look at the rate, which takes into account the subreddit growth, not just a noticeable increase in threads, but a veritable explosion of users, there is something of a decline.

It shouldn't really come as too much of a surprise to see. In the past, doing flair drives and discussing engagement with the academy, we've talked about how it can be an uphill battle. We've incredibly proud of the large panel of flaired users we have contributing to the site, not to mention countless more users who dive in to share their knowledge even if not members of the panel, but growth can be hard to maintain consistently, and this helps to illustrate that while the subreddit continues to grow, the contributor base is not able to match that pace. Controlled growth is one of the largest reasons we have always declined default status in the past, or being included in the new onboarding menu that replaced the defaults, but even with the slower growth that remaining outside provides, new blood into the ranks of contributors is vital, and something that we are always trying to improve on.

The other thing that I think is very noticable is that while the Response per Day stayed pretty consistent - again, 46.95 compared to 47.61 - the Answers per Day did drop noticeably, the 2017 numbers being 41.56 compared to 2016's 44.32. While this might, at first, seem unfortunate, I feel that it really reflects the nature of the subreddit. Some questions are new and original, some are retreads which might still offer new angles, but many of course are retreads. And while our philosophy is that no answer is ever definitive, the accumulation of answers does mean that some of the more common questions do become more and more likely to have a linked response rather than a new answer, so a divergence over time between the Response Rate and the Answer Rate should be fairly expected, a simple reflection of the ever accumulating base of knowledge that exists on the subreddit. It is fair to say that the divergence would be even greater is links/reposts by the author of the original were counted in the Response Rate only and not as Answers, but unfortunately not a statistics I have tracked, so we can only speculate on the exact impact, although I would venture at least more than one per day.

Finally, of course, looking at the Response Rate of 40.5 across the past two years, one other thing seems worthy of note. Obviously, in a perfect world, every single question would get the response it deserves, but that can't be in the cards. To be sure, we would be happy to see it rise a bit more, but we also know that there is an upper limit to what is possible, and sacrificing quality for quantity simply isn't the compatable with the purpose of the subreddit. Even were we to see considerable gains in the number of flaired users, the simple fact is that the time and effort it takes to provide a response guarantees will always mean that we see a lower response rate than in other subreddits.

By way of comparison, if we look at /r/ExplainLikeImFive, or /r/AskHistory, subreddits similarly organized around the "Ask Questions, Get Answers" format, but with a much lower bar for what it allowed and what isn't, you certainly have a better chance of a response. But they offer different experiences, ones that we encourage users to try out if they aren't looking for what we have on offer here. /r/AskHistorians isn't where you come for any old response, it is where you come for a specific type of response. So while we'd love to see that rate rise, we also are cautious about how much us simply possible, and generally pleased to see it where it is.

That said, it isn't easy to say what the upper bound is, but if I had to guess, I doubt that even with serious expansion of the flaired user base we could expect to rise much beyond 50 percent response rate, but there is very little good basis for comparison. I actually attempted to do so by using ELI5, pulling the complete submission list for the month of November to look at comparables, but was somewhat surprised by the results! On the one hand, of the 2901 approved submissions that month, only 307 recieved zero comments, an "Ignore Rate" of merely 11 percent. That is hardly a perfect metric, since while their rules are obviously different, there still are comments which get removed, such as in this thread with one comment which was removed. And further of course, answers which are allowed can often lack the depth and accuracy that AskHistorians requires (I don't want to put anyone on the spot by highlighting an example, as I don't mean this as a criticism, but anyone familiar with these two subreddits should know the differences I speak of).

But what I found quite fascinating was that ELI5 actually had 15,487 submissions in November, with 12,586 submissions showing as [removed] in the dataset. Again, not a criticism of their sub, as you won't find any mod team more understanding of a team's desire to ensure things run smoothly, but I did find it illustrative of their somewhat different approach. To be sure, I did not check a large number of the removed submissions, but one thing that did strike me was their seemingly strong enforcement of Rule 7, "Search before posting; don't repeat old posts", as in the random spot-checking I did, I encountered several removals for that reason (plenty more, however for Rule 2). As regulars are well aware, we do maintain an FAQ, but we have no ban on repeat questions. Our policy allowing linking to old answers is intended to help carry the weight there, but nevertheless, having stared at literally tens of thousands of questions asked on this sub over the past few years, I can easily say that many do get asked for which there is almost certainly a previous answer on the subreddit, but which never get linked to it (Allow me to plug our FAQ Finder Flair!).

So anyways, the point of this digression is to help illustrate that different approaches lead to different results, and both have their strengths and weaknesses. Using one example of this, ELI5, being a larger subreddit than us, having roughly 5x as many submissions per month than we do, decided that "Asked and Answered" questions ought to be removed, a perfectly reasonable decision given the impact such volume has on distribution of mod 'resources'. In turn, we don't take such an approach, but maybe it is one we would do if we were getting 15,000 questions a month! Doing so would likely seriously impact the Answer Rate in a positive direction, but it isn't something we are interested in doing, as we don't find that it is an undue impact on our time to allow them, and as has been said before, we consider no response, even a 19 comment answer by /u/pangerandipanagara, to be the absolute final word. There are other moderation decisions we make that likewise impact how the subreddit functions, and likewise we have our reasons, but to get back to the original point at hand, the end goal is always being conducive to the fostering of high quality content on the subreddit, and we will always value quality over quantity.

I don't have anything specific to mention here. Seeing the stats for each given day helps to illustrate how varied the numbers can be, but I'm not sure how useful they really are to look at!

So there we have it. Part II will hopefully be forthcoming soon. I have data for most of the year, which I've been playing around with, but still am waiting on the December stats. In any case, it will offer some other insights which I hope some will find interesting!

Footnotes:

1. Response Rate: The percentage of questions which receive a response of either an answer, or a link to a previous thread or FAQ section. Other visible responses such as follow up questions
2. Answer Rate: The percentage of questions which receive an answer, excluding responses which link to previous threads or the FAQ, except in cases where it is the original author linking.
3. Times: These are for the first visible answer that appeared. This excludes comments which are links, and does not factor questions which remained unanswered. Average excludes outlier threads where the answer was >48 hours after posting. Minimum and maximum only note cases where there was an answer, not a link.
4. Total: These are estimates for the month, based on surveys of seven semi-randomly chosen 24 hour periods with all threads in that period checked, excluding META and Feature threads from the count. Each day's value is provided, and then the rate for the combined.
5. Insufficient: This is the questions which did receive replies, but either none remain visible, or else what is visible is not an attempt to answer the question, such as mod warnings, or unanswered follow-ups.
6. Ignored: This covers questions which received no comments at all, visible or otherwise. It also does not make any judgement on whether the question was answerable, or well phrased.
7. Days: These are chosen with a random number generator, with discretion to exclude US Federal Holidays, as these are likely to reflect abnormal traffic and usage patterns. One of each day of the week is chosen, i.e. Monday, Tuesday, etc, with an avoidance of consecutive days, and at least one day for each week of the month. Weekends are in italics.

This twitter thread (https://mobile.twitter.com/ireland/status/954792642327523329) lists a number of accusations against Winston Churchill. The poster accepts that he is no historian, but many of his accusations seem accurate to a casual reader. How reliable are his claims?