12
$\begingroup$

Is there a way to answer questions like the following, either by means of the StackExchange Date Explorer or by other means?:

  • How many of the regular MathOverflow users (say, e.g., those who contributed at least $n$ posts with positive score) live on which continent / in which country?

  • How many new user's first questions have been asked during the last 12 months from which continent / country?

The "Location" field in the StackExchange Date Explorer doesn't seem suitable for this, as it is often left blank or contains text which is hard to interpret automatically (the city, the province, the university, etc.).

$\endgroup$
3

2 Answers 2

Reset to default
10
$\begingroup$

Maybe by other means...

The public interface to the database is sanitized to protect user privacy (see the privacy policy). Personal information that could be used to identify an individual is removed or anonymized. This is especially true for sensitive personal information that could be used to reveal race, ethnic origin, political opinions, religious or philosophical beliefs, concerns health or sex life, criminal and legal information, etc. Excluded from this is any such information that is volunteered by the user, such as the optional location and age fields on the user page. Since location does classify as sensitive personal information, if a user does not provide that information in the location field of their user page, you will not find it in the public database.

That said, the restricted database does keep track of IPs and other info that is sufficient to resolve the location of a user to a certain degree of accuracy (unless they are using anonymizing service). If you explain your reason for obtaining this information, you convince appropriate team members that this is a good idea, and the information requested does not violate the privacy policy and other applicable laws, then there is a chance that the database keepers will take the time to accommodate your special request and gather the data you need.

$\endgroup$
1
  • $\begingroup$ Well -- I asked the question for mere reasons of curiosity, and because I thought some statistics on the geographical distribution of the MO community might be of interest also for others. Note also that we are talking here about statistics, and not about data of individual people. $\endgroup$
    – Stefan Kohl Mod
    Commented Jan 25, 2015 at 20:52
9
$\begingroup$

There are a number of scripts on data.stackexchange.com that give information about users by country.

For example, the World Cup of Reputation tells you the number of users and total reputation of users by their self-reported country, and then sorts the list by the average of the top 100 per country. (It appears that the script could be improved, however, since it finds Noah Synder to be the top user from India, whereas I believe he is from Bloomington, Indiana.)

There are a large number of scripts accessing the location field, and you could play around with them to get what you want from it.

$\endgroup$
12
  • 6
    $\begingroup$ You mean Indiana is not in India? $\endgroup$
    – Gerald Edgar
    Commented Jan 26, 2015 at 15:04
  • 5
    $\begingroup$ Well, looking at the SQL query that's explainable: if the location has "india" as substring, it matches. More worrisome: any location that contains ", " matches the US! $\endgroup$
    – Jaap Eldering
    Commented Jan 26, 2015 at 15:57
  • $\begingroup$ These scripts are certainly interesting (thanks for referring to them!), but the quality of results is pretty limited - mainly for the reason that many (most?) users left the 'Location' field blank, and also (as @JaapEldering has already pointed out) for the reason that the script often doesn't interpret the 'Location' entries correctly where they are provided. $\endgroup$
    – Stefan Kohl Mod
    Commented Jan 26, 2015 at 16:37
  • 1
    $\begingroup$ @JaapEldering: More precisely, any location that doesn't match any earlier cases and contains ", " is matched to the US. The idea is to catch "New York, NY" and "Hanover, NH". A fancier script could match all 50 state abbreviations explicitly. $\endgroup$
    – François G. Dorais Mod
    Commented Jan 26, 2015 at 19:17
  • 1
    $\begingroup$ There are dozens of scripts there, many of them spinning off from each other, and you can fork your own and make revising edits. Here is a version of the world cup script data.stackexchange.com/mathoverflow/query/78712/… that has a better treatment of US states, but it still falls prey to the Indiana/India conflation, which presumably could be fixed by adding a single additional line. I'd encourage anyone who cares about the data to start writing their own scripts. $\endgroup$
    – Joel David Hamkins
    Commented Jan 26, 2015 at 22:15
  • $\begingroup$ Here is a version with the Indiana issue fixed: data.stackexchange.com/mathoverflow/query/267996/… $\endgroup$
    – François G. Dorais Mod
    Commented Jan 27, 2015 at 2:44
  • 3
    $\begingroup$ I'm afraid some of the data linked in Joel's answer may not have caught up with historical developments: Ireland, for example, appears there to be part of the United Kingdom. (`WHEN UPPER(Location) LIKE '%IRELAND%' THEN 'United Kingdom') $\endgroup$
    – shane.orourke
    Commented Jan 27, 2015 at 11:33
  • $\begingroup$ @shane.orourke It would be kind of you to fix that issue --- it seems to me you can just change it to: THEN 'Ireland', and you might add some lines for Dublin, Galway and Cork, for example. $\endgroup$
    – Joel David Hamkins
    Commented Jan 27, 2015 at 12:33
  • 1
    $\begingroup$ @JoelDavidHamkins I've made a few modifications along these lines. The new version is at data.stackexchange.com/mathoverflow/query/268646/…. $\endgroup$
    – shane.orourke
    Commented Jan 28, 2015 at 21:24
  • $\begingroup$ @shane.orourke Thanks! But your version still has the Indiana/India issue, and it appears that you didn't start from the script treating the US states properly... $\endgroup$
    – Joel David Hamkins
    Commented Jan 29, 2015 at 14:44
  • $\begingroup$ The special cases in that script are pretty amusing. Among places that are specifically detected as part of the United States: COUMBIA SC [sic], HAWAI [sic], HOLLYWOOD, and that bustling metropolis, QUAD CITIES. $\endgroup$
    – Nate Eldredge
    Commented Feb 2, 2015 at 19:29
  • $\begingroup$ @NateEldredge That's funny! I had always assumed that the cases in those scripts were drawn mainly from actual cases in the (SO) database that were observed to arise, but now I'm not so sure... $\endgroup$
    – Joel David Hamkins
    Commented Feb 2, 2015 at 21:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .