13
$\begingroup$

One way to measure the effectiveness of MO is to track how often it leads to publishable research. That angle has been discussed before.

I'm wondering if there are other metrics that can be used to assess the effectiveness of MO. For example, how many questions get asked? How many get answered? Has this changed over time? Is there any evidence that "soft" questions have a beneficial effect by attracting more users, who then see the "hard" questions—or is there any evidence of the opposite, that "soft" questions cause users to abandon MO in disgust?

If there aren't such metrics, is there a way to create them? Or perhaps it's a bad idea to create such metrics, because they will be misinterpreted, leading to bad side-effects?

$\endgroup$
8
  • 6
    $\begingroup$ I'm a bit unclear what you're asking. Are you asking how to measure certain things (e.g. how to confirm or debunk ideas about the effect of soft questions)? Or are you asking whether certain kind of raw data is (or could be) available to the general public? Or are you asking whether such measurements have been done (but kept quiet for some reason)? $\endgroup$ Mar 2, 2016 at 6:49
  • 1
    $\begingroup$ Moderators and 25k+ users have handy access to some data. (I do not know if 25k+ users have the same data or a subset.) Quite a few things can be obtained from the Data Explorer too. For how many questions get asked there is a dedicated question on that On MESE we sometimes have huge (in a relative sense) peeks of visitors (think at least an order of magnitude above the usual level). As far as I can tell "with my bare eyes" (I did not analyse this).I cannot make out any signfiicant sustained effect. $\endgroup$
    – user9072
    Mar 2, 2016 at 10:29
  • $\begingroup$ @FrançoisG.Dorais : Of your three options, my answer to #2 is yes; my answer to #1 is yes but it's contingent on the raw data being available so it's a secondary question; my answer to #3 is no, that idea hadn't crossed my mind. $\endgroup$ Mar 2, 2016 at 15:39
  • 2
    $\begingroup$ Another "data thread" is the 'Best of MO'. $\endgroup$
    – user9072
    Mar 2, 2016 at 16:06
  • $\begingroup$ Regarding change over time, I collected some numbers and searches in this "birthday" post: meta.mathoverflow.net/a/2486 $\endgroup$
    – j.c.
    Mar 2, 2016 at 17:49
  • $\begingroup$ There are also some site analytics available to rep>25k users (via /site-analytics). It is interesting enough for getting a rough idea of traffic volumes, patterns, and sources, but it's also relatively limited as a tool. $\endgroup$ Mar 9, 2016 at 18:36
  • $\begingroup$ In addition to site analytics mentioned in the previous comments, some statistics can be found also in the list of all sites. (These are only very basic stats, but they are not restricted to 25k+.) $\endgroup$ Mar 28, 2019 at 7:18
  • $\begingroup$ These posts suggest some statistics, too: Development of the MathOverflow community and What statistics would be interesting for a usage report? $\endgroup$ Jan 6 at 12:28

2 Answers 2

15
$\begingroup$

A lot of MathOverflow data is already available publicly.

An anonymized version of the entire Stack Exchange Network database is archived every few weeks and permanently on archive.org. It's easy to download just the data for MathOverflow, which is currently 189.4M compressed. That data consists of 7 xml files (badges.xml, comments.xml, posts.xml, posthistory.xml, postlinks.xml, users.xml, votes.xml) whose contents are described in this readme file. This is essentially the full site database, minus personally identifiable information (PII). The data is licensed under the usual CC BY-SA 4.0 license used for the site itself.

For those who would rather avoid downloading the entire database, the data is also available from the Stack Exchange Data Explorer which provides an interface to make SQL queries to the database. Another alternative is to use the Stack Exchange API. All of these options have pros and cons depending on what kind of processing is to be done with the data.

For security reasons, very very few people have access for the full database which includes PII and other sensitive data. However, these select individuals have been known to be helpful and process reasonable queries that would be impossible with the anonymized database.

$\endgroup$
0
3
$\begingroup$

Certainly not all of them, but at least some of the questions you're asking can be answered using SEDE. (I will admit, the you probably do not find here answers to the questions which are most interesting to you; such as why people join or leave; when MO posts lead to published results, etc.)

SEDE was already mentioned in Fran├žois G. Dorais' answer - let me add at least some queries corresponding to some of the questions you asked.

How many questions get asked?

Many queries concerning number of question and answers can be found here: Does the number of answers really drop over years? It would not make much sense to repeat all of them here - so I will just include a graph showing the number of questions/answers/posts per month.

Graph

This only includes the posts that haven't been deleted - if you want to see stats with deleted posts, you can find such queries in the linked post.

Definition of answered question

One could take various definitions of when the question is considered answered. Does it have at least one answer? Does it have an accepted answer? Does it have at least one answer with a positive score? (You probably know that for the purposes of various lists of unanswered questions, Stack Exchange uses the definition that a question is answered if it has an accepted answer or an upvoted answer.)

For the simplicity, I will count the questions where at least one answer was posted and the question which have an accepted answer.

You can find some stats about answers also here: Does the number of answers really drop over years?

How many get answered?

We can look at the number (and at the percentage) of questions which got at least one answer. Naturally, it is expected that older questions should have a bit higher percentage - since there was a longer time for potential answerers to think about the problem and post an answer.

To clarify, I will explicitly mention that the query is looking only at the question posted in the given month. (For example, in December 2023 we see the value around 50%. That means that half of the questions posted in December 2023 already had at least one answer when the query was run. So this is not saying that only half of all questions had an answer at the time.)

This is how the graph with the percentage looks at the moment:

Graph

If we exclude closed questions from our stats, the graphs do not look very differently: numbers and percentages.

How fast they get answered?

If you look at the time until the first answer, the majority of questions gets an answer on the same day as the question is posted and the majority of answers is posted on the same day as the question.

In connection with this, one might be interested to see questions the longest and the shortest time until the first answer. For the shortest time, it make sense to exclude the answers posted simultaneously with the question (and the answers which are older than the question - such answers are probably consequence of merging two questions). Or look at the first answer posted by somebody else than the asker.


Feel free to ask if you think some other SEDE queries might be interesting for you - either in comments or in chat. (For example, it is some work - but it is not terribly difficult to create similar queries which look at the same stats per year rather than per month.)

Chat is probably better - it is more suitable for longer exchanges, with an explanation what can and cannot be done in SEDE. A good place might be one of the SEDE chatrooms - here and here. Or, if you prefer, you can ask me directly in my chatroom.

$\endgroup$
1
  • $\begingroup$ I find the percentage-of-questions graph remarkable, surely a decline not fully explained by the more time to answer older questions. I wonder what this portends for the future of MathOverflow... $\endgroup$ Jan 8 at 22:49

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .