24
$\begingroup$

One of the more controversial aspects of the transition to MO2.0, for myself at least, was the reduction in the reputation added for upvotes on questions asked from +10 to +5, while retaining the reputation for answer upvotes at +10. I doubt there is any way to change this, but I would be interested in whether we can establish its effect.

There is a school of thought (to which I subscribe) that holds that the purpose of answers is to ask better questions, and questions rather than answers are primary. Voltaire, for example, said "Judge a man by his questions rather than by his answers".

For myself, on a completely personal level, I have noticed that I am less motivated now to ask well thought-out questions because they are "worth less". Is this experience shared? Can it be quantified?

Also, the research of Yla R. Tausczik and collaborators correlated high "real-life reputation" with good questions, but not necessarily with good answers. Again, are these further statistics to back this up?

In particular, interesting statistics might include:

  1. Do people ask better questions when they get more reputation for an upvote (relative to answer reputation)? Can this be quantified in a useful way?
  2. How strong is the correlation noted by Tausczik et al. between real-life reputation and question quality?
  3. Are good answers leading to better questions? When MO answers are motivated by MO questions, are they better questions if the former questions have better answers? Do highly upvoted answers tend to lead to questions?
  4. Is mathematics special in regard to the above, or would the same conclusions hold for other fields of human knowledge?
  5. Which other statistics, besides those mentioned above, might be used to test the philosophical/ sociological theory that "a good question is better than a good answer" (maybe this is really a question for sociology Q&A site (^_^) ).
$\endgroup$
18
  • 2
    $\begingroup$ Personally, I believe the impact of points on the quality of questions is probably minimalistic. Also, if there is such effect, how would you quantify it -- maybe by numbers of upvotes (where voting behavior of people may also depend on points), or how else? $\endgroup$
    – Stefan Kohl Mod
    Mar 30, 2015 at 16:05
  • 4
    $\begingroup$ I doubt there is an impact. People are going to ask their good questions regardless of the points, because they're interested in answers, not because they're interested in points. If I think of the questions I have that I have not yet asked (or have not been asked by other people) they're all either too technical or difficult, boring or controversial. Maybe the controversial ones could be massaged into good questions, but it's not points that will motivate me -- only spare time would motivate me. $\endgroup$ Mar 30, 2015 at 21:12
  • $\begingroup$ Maybe number of upvotes; it might also be interesting to look at number of upvotes weighted by reputation of upvoter. Another thought would be number of views, although this would be biased towards questions with wide appeal. $\endgroup$ Mar 30, 2015 at 21:50
  • $\begingroup$ Another thought- number of characters in a question, the concept being that length of question correlates positively with quality. $\endgroup$ Mar 30, 2015 at 21:55
  • 9
    $\begingroup$ You think that "length of question correlates positively with quality"? -- I doubt. -- Both good and bad questions can have various lengths, and very long questions are often low-quality (a.k.a. "too broad" or "unclear what you're asking") in my experience. $\endgroup$
    – Stefan Kohl Mod
    Mar 30, 2015 at 22:17
  • 1
    $\begingroup$ Stefan Kohl: My thought is that longer questions evidence that more work has gone into them, so while bad questions might certainly be long, I would wonder whether there is a positive correlation between length and quality. In any event, +5 leading to shorter questions would provide evidence, I think, that it reduces effort put into questions, and it leading to longer questions would evidence the opposite. $\endgroup$ Mar 30, 2015 at 22:23
  • $\begingroup$ I don't understand that. Do you ask questions to gain reputation or do you ask question to gain insights from the answers? I can attest that I ask less questions recently, but that's because nowadays I'm working on my Ph.D. and my ideas are such that I really have no one to ask. So most of my questions get discussed with my advisor, rather than on MathOverflow. But the reputation? That was never a factor. (Although to the record, I do agree that at least on MO, questions should be worth as much as answers. And I did protest this change before the migration, several times.) $\endgroup$
    – Asaf Karagila Mod
    Mar 31, 2015 at 0:05
  • 4
    $\begingroup$ There is a another asymmetry between questions and answers that I have wondered about: questions can get votes to close while answers don't. $\endgroup$ Mar 31, 2015 at 12:41
  • 1
    $\begingroup$ Dear Daniel, I am not sure that the statistics supports your personal reaction as you described it because the ratio of questions you asked since the change is not smaller (perhaps even larger) than the ratio of questions you asked after the change. (This applies to me even more strongly.) $\endgroup$
    – Gil Kalai
    Mar 31, 2015 at 21:27
  • $\begingroup$ Dear Gil: But maybe not their quality ;-) Also, there's an interesting correlation which I would hypothesize here: When somebody asks a question, they then check the site more often for a certain period, making them also more likely to answer questions. Thus, asking a well thought-out question might actually decrease the ratio of questions to answers! Is this true? Could such a hypothesis be tested? (straying deep into sociology/ behavioural psychology...) $\endgroup$ Apr 1, 2015 at 7:41
  • 2
    $\begingroup$ There are some (rather not me though) that think many of the best questions are among those that by tradition are turned CW and then yield no points. In my mind this makes your theory quite doubtful. $\endgroup$
    – user9072
    Apr 1, 2015 at 10:20
  • 2
    $\begingroup$ Thanks Daniel and quid for the lovely responses and interest. Of course, I meant "before" and not "after." Usually my writing can be characterized by being acted by a large $(Z/2Z)^m$ group of errors. Quid, I dont think the CW effect makes a difference regarding my theory (we can check the statistics without CW questions and answers).But actually I did not propose a theory but rather some statistical concerns (or counter arguments) regarding Daniel's theory :) . $\endgroup$
    – Gil Kalai
    Apr 1, 2015 at 19:51
  • 5
    $\begingroup$ I'm honestly shocked to hear you're 'less motivated now to ask well thought-out questions because they are "worth less".' I myself am motivated to ask a question when I care about the answer, and MO is the best place to ask it. Then I make it well thought-out (to within my ability) the better to attract answers. Once my reputation got high enough to be able to correct other people's spelling mistakes, I stopped caring about scoring. $\endgroup$ Apr 2, 2015 at 1:51
  • 2
    $\begingroup$ Allen Knutson: Let me further clarify the distinction. Let's say I want to know whether X satisfies A. If I think I might get an answer, but without external reinforcement (points), I might ask "does X satisfy A?" Only experts would understand. If I got lots of external reinforcement, I might write a question saying what X is, why we would care if X were to satisfy A and what it would imply, and then the question. It wouldn't effect the quality of the answer, perhaps, but it would be a better question for the non-expert. If I'm honest with myself about my motivations, MO rep matters for nonCW. $\endgroup$ Apr 2, 2015 at 8:54
  • 3
    $\begingroup$ (To complement Voltaire.) Cantor: In re mathematica ars proponendi quaestionem pluris facienda est quam solvendi. Einstein: The formulation of a problem is often more essential than its solution... To raise new questions, new possibilities, to regard old questions from a new angle, requires creative imagination and marks real advances in science. And Wertheimer: Often in great discoveries the most important thing is that a certain question is found. Envisaging, putting the productive question is often more important, often a greater achievement than solution of a set question. $\endgroup$ Apr 2, 2015 at 17:43

2 Answers 2

30
$\begingroup$

I've looked at some related questions on Stack Overflow, Super User and Server Fault. On those sites, I found no evidence that voting patterns changed as a result of the reputation payout. Given the unique history of this site, it's interesting to see if we can see something different here. I'm taking June 25, 2013 as the transition date. As with the other sites I looked at, there's no obvious change to the way people vote on questions compared to answers before and after the change:

MathOverflow voting trends

Meanwhile, there is a strong difference in the number of downvotes cast on questions due to more votes being available and removing the reputation penalty for downvoting questions:

Downvoting on questions

(Note: I link to a query that excludes deleted posts, but the graph shown includes them. For obvious reasons, questions liable to be deleted tend to attract more downvotes so the public data is noticeably different. Even so, there is an obvious increase in downvoting after the change in system.)

Using the methods I used to look at the original trilogy, I don't see much evidence that people ask more or better questions when offered 10 reputation per upvote compared to just 5. However, there is strong evidence that people are more free with their downvotes on questions now that they don't cost reputation and more of those votes are allocated.

Your observation that you are less motivated by 5 reputation compared to 10 is interesting. Another pattern that MathOverflow users share with users on other sites is a clear tendency to shift from asking to answering questions as they gain experience. I'm not sure why that is, but I think it's a result of intrinsic motivations. Most people have a finite set of interesting and difficult questions that they've considered over their experience with a topic. When they find a Q&A site, they naturally are motivated to ask those questions. As they receive answers and run out of questions to ask, people naturally get excited about helping others with their interesting and difficult questions. I suspect for many people the relative payout of questions to answers is not a large factor if it is a factor at all.

Psychological research strongly suggests that people do better work when motivated by something inside of themselves (such as curiousity) than when they are motivated externally (such as by votes and reputation). Therefore we should expect that increasing the reputation payout will have no (or even negative) effects on average question quality.

Lastly, I agree with your philosophical point: good questions are worth more than good answers. However, you have to use the correct scale. As the asker, an answer can be invaluable. Q&A communities, on the other hand, need a constant supply of compellling questions to stay vital.

$\endgroup$
5
  • 2
    $\begingroup$ Am I correct in thinking that the vertical axes on these graphs are the difference between the total number of question votes and the total number of answer votes made each week (the first graph being votes of any kind, and the second being only downvotes)? $\endgroup$ Mar 30, 2015 at 20:01
  • $\begingroup$ @EricWofsey: Precisely. $\endgroup$ Mar 30, 2015 at 20:02
  • 2
    $\begingroup$ Thanks! This is a great answer!! My follow-up question would be whether numbers of upvotes are the correct metric to use (certainly it is the most obvious). Other options might be number of views per question, or number of characters per question (are both positively correlated with "quality"? I don't know). Another metric might be decay rate of upvotes with time. Do we see a different pattern with respect to these other metrics? $\endgroup$ Mar 30, 2015 at 22:04
  • 4
    $\begingroup$ "Psychological research strongly suggests that people do better work when motivated by something inside of themselves than when they are motivated externally". I think we should be very skeptical about such statements. However, one should be aware that the effects of the external motivations and incentives (and certainly internal ones) can be quite subtle. $\endgroup$
    – Gil Kalai
    Apr 4, 2015 at 19:02
  • 1
    $\begingroup$ @Gil Kalai: The research is very compelling that extrinsic motivation can drive out intrinsic motivation in certain circumstances. Stack Exchange sites are token ecomonmies which can be an exception. Receiving upvotes tends to be a more powerful reinforcement than gaining reputation. $\endgroup$ Apr 6, 2015 at 7:12
8
$\begingroup$

You wrote:

For myself, on a completely personal level, I have noticed that I am less motivated now to ask well thought-out questions because they are "worth less". Is this experience shared? Can it be quantified?

This is very surprising. I hope that this does not apply to most participants. For myself, the reputation on this site is completely irrelevant. I am here not to gain reputation, but to communicate with other mathematicians.

One my friend whom I proposed to join this site said that the very existence of "reputation points" is something he does not like about this site. But I understand that reputation points can be useful, for example those who have certain amount can have some moderator privileges. This seems reasonable.

And on my opinion, the exact formula how reputation points are awarded is completely irrelevant, if its purpose is to distinguish those who actively participate in this cite, and make positive contribution.

Another point: I disagree with the statement (which you seem to consider evident) that "good question" or "good answer" is that one which has many up votes. Look at the questions which have maximal number of up votes. Most of them are "soft questions". The reason is simple: more people understand them.

Third point: you refer to a research paper which deals with "real-world reputation". What is this and how is it measured? I looked at the paper and used the search: on "real-world" I found nothing. On "reputation" I found only 5 occurrences and all of them were about MO reputation.

$\endgroup$
3
  • 3
    $\begingroup$ Dear Alexandre, Thank you for this. I am interested though, in data: I would argue that the StackExchange concept would not work without "reputation". If the site gave no reputation, I doubt the concept would take off, just in terms of sociology. If reputation is an (irrational) motivator, then, it makes sense to ask how RELATIVE reputation effects motivation. Admittedly this is a question for a cognitive psychologist more than for a mathematician, but we can certainly ask it. $\endgroup$ Apr 3, 2015 at 8:11
  • 4
    $\begingroup$ Reputation seems to be one of the major driving factors of Stack Exchange. I would hesitate to decline that. $\endgroup$
    – Jinxed
    Apr 3, 2015 at 14:58
  • 6
    $\begingroup$ I agree that reputation, and other forms of formal and informal recognition are surprisingly powerful driving factors in MO and in this respect as others, MO is a nice microcosmos which is quite telling about the real world. $\endgroup$
    – Gil Kalai
    Apr 4, 2015 at 17:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .