Is there a way to measure the effectiveness of MO?

Question

One way to measure the effectiveness of MO is to track how often it leads to publishable research. That angle has been discussed before.

I'm wondering if there are other metrics that can be used to assess the effectiveness of MO. For example, how many questions get asked? How many get answered? Has this changed over time? Is there any evidence that "soft" questions have a beneficial effect by attracting more users, who then see the "hard" questions—or is there any evidence of the opposite, that "soft" questions cause users to abandon MO in disgust?

If there aren't such metrics, is there a way to create them? Or perhaps it's a bad idea to create such metrics, because they will be misinterpreted, leading to bad side-effects?

Answer 1

A lot of MathOverflow data is already available publicly.

An anonymized version of the entire Stack Exchange Network database is archived every few weeks and permanently on archive.org. It's easy to download just the data for MathOverflow, which is currently 189.4M compressed. That data consists of 7 xml files (badges.xml, comments.xml, posts.xml, posthistory.xml, postlinks.xml, users.xml, votes.xml) whose contents are described in this readme file. This is essentially the full site database, minus personally identifiable information (PII). The data is licensed under the usual CC BY-SA 4.0 license used for the site itself.

For those who would rather avoid downloading the entire database, the data is also available from the Stack Exchange Data Explorer which provides an interface to make SQL queries to the database. Another alternative is to use the Stack Exchange API. All of these options have pros and cons depending on what kind of processing is to be done with the data.

For security reasons, very very few people have access for the full database which includes PII and other sensitive data. However, these select individuals have been known to be helpful and process reasonable queries that would be impossible with the anonymized database.

Answer 2

Certainly not all of them, but at least some of the questions you're asking can be answered using SEDE. (I will admit, the you probably do not find here answers to the questions which are most interesting to you; such as why people join or leave; when MO posts lead to published results, etc.)

SEDE was already mentioned in François G. Dorais' answer - let me add at least some queries corresponding to some of the questions you asked.

How many questions get asked?

Many queries concerning number of question and answers can be found here: Does the number of answers really drop over years? It would not make much sense to repeat all of them here - so I will just include a graph showing the number of questions/answers/posts per month.

This only includes the posts that haven't been deleted - if you want to see stats with deleted posts, you can find such queries in the linked post.

Definition of answered question

One could take various definitions of when the question is considered answered. Does it have at least one answer? Does it have an accepted answer? Does it have at least one answer with a positive score? (You probably know that for the purposes of various lists of unanswered questions, Stack Exchange uses the definition that a question is answered if it has an accepted answer or an upvoted answer.)

For the simplicity, I will count the questions where at least one answer was posted and the question which have an accepted answer.

You can find some stats about answers also here: Does the number of answers really drop over years?

How many get answered?

We can look at the number (and at the percentage) of questions which got at least one answer. Naturally, it is expected that older questions should have a bit higher percentage - since there was a longer time for potential answerers to think about the problem and post an answer.

• Here is the number of questions having at least one answer (or an accepted answer) per month.
• And here is the percentage of such questions among all questions

To clarify, I will explicitly mention that the query is looking only at the question posted in the given month. (For example, in December 2023 we see the value around 50%. That means that half of the questions posted in December 2023 already had at least one answer when the query was run. So this is not saying that only half of all questions had an answer at the time.)

This is how the graph with the percentage looks at the moment:

If we exclude closed questions from our stats, the graphs do not look very differently: numbers and percentages.

How fast they get answered?

If you look at the time until the first answer, the majority of questions gets an answer on the same day as the question is posted and the majority of answers is posted on the same day as the question.

In connection with this, one might be interested to see questions the longest and the shortest time until the first answer. For the shortest time, it make sense to exclude the answers posted simultaneously with the question (and the answers which are older than the question - such answers are probably consequence of merging two questions). Or look at the first answer posted by somebody else than the asker.

---

Feel free to ask if you think some other SEDE queries might be interesting for you - either in comments or in chat. (For example, it is some work - but it is not terribly difficult to create similar queries which look at the same stats per year rather than per month.)

Chat is probably better - it is more suitable for longer exchanges, with an explanation what can and cannot be done in SEDE. A good place might be one of the SEDE chatrooms - here and here. Or, if you prefer, you can ask me directly in my chatroom.