Please be more careful when interpreting the Stack Overflow Developer Survey • Hacker News Board

Re: Please be more careful when interpreting the Stack Overflow Developer Survey

#11

Post by prepend » Sat, Apr 13, 2019, 3:20 PM UTC

shadowmint wrote: post #4

> you cannot generalize from a non-random sample So, honest question: If any survey of any size can be ignored on the basis that the sample is not random, then how is any survey meaningful? Isn’t this a self defeating argue? You can’t prove the sample is random, all you can do is show differences between samples and suggest its not consistent ... but how do we go away and prove that some other survey we’re comparing…

The key is in how you randomly select the sample from the population.

This was the author’s point. Just because you have 90k SO respondents doesn’t mean you can say anything about developers as a population. You can say lots of stuff about SO users. Or maybe developers who use SO. But just because you have lots of responses doesn’t mean you know what developers or jugglers or farmers or whatever population interests you.

The confusion rests with SO’s statement that their survey should be representative of developers in general (or CS graduates or whatever other than only SO visitors).

Re: Please be more careful when interpreting the Stack Overflow Developer Survey

#12

Post by MaulingMonkey » Sat, Apr 13, 2019, 3:20 PM UTC

shadowmint wrote: post #4

> you cannot generalize from a non-random sample So, honest question: If any survey of any size can be ignored on the basis that the sample is not random, then how is any survey meaningful? Isn’t this a self defeating argue? You can’t prove the sample is random, all you can do is show differences between samples and suggest its not consistent ... but how do we go away and prove that some other survey we’re comparing…

> If any survey of any size can be ignored on the basis that the sample is not random, then how is any survey meaningful?

One can take efforts to make the sample more random. This is part of the reason why the U.S. Census is legally compelled, for example - to try and reduce self-selection bias. Or the push for mandatory standardized tests in schools.

One can contextualize the results. Applying, say, English literacy rates from a U.S. Survey to China is obviously going to be totally wrong. Applying a developer salary survey at Google to Game Developers is going to be totally wrong. But within their context, they can be more accurate. Outside of their original context, the survey can be re-run.

> ie. Isnt this just a convenient excuse to deny that a survey is meaningful?

While convenient, it's sometimes also inconveniently true that a survey isn't terribly meaningful, or isn't in the context it's being reapplied in. Statistical stuff is hard, a lot of surveys are bad, and while you can make some reasonable guesses and extrapolations, it's worth doing so with a giant grain of salt.

Re: Please be more careful when interpreting the Stack Overflow Developer Survey

#13

Post by shkkmo » Sat, Apr 13, 2019, 3:31 PM UTC

shadowmint wrote: post #10

ChrisSD wrote: post #6

Earlier quoted context omitted.

> I don’t accept you can survey 90000 developers and cannot offer any generalisation from those results without quanatitively proving there is an overwhelming sample bias, and specifically quantifying the degree of that bias. Surely you have this backwards? If you want to argue that a survey offers any generalisation, then surely the onus is on you to prove you've accounted for sample bias (amongst others)?

That seems fair; but they have a whole methodology section. If you want to argue with it, surely the onus is on you to do it concretely? > Because of your methodology, we must assume a biased sample. ^ I find this quote problematic. Why must we assume that? If you want to distribution comparisons and point out there survey results are skewed by X compared to some other survey Y... ok. ...but that’s not whats happenin…

> Why must we assume that?

Because you should distrust flawed methodologies by default. The incorrect assumption is that the sample produced by a known flawed methodology is representative.

> Its just a flat out arbitrary assumption.

It is not at all arbitrary. It is based on well known issues with this particular method of sampling.

Re: Please be more careful when interpreting the Stack Overflow Developer Survey

#14

Post by _pastel » Sat, Apr 13, 2019, 3:32 PM UTC

Next year, ask: "Compared to developers of similar position, experience, and willingness to respond to Stack Overflow Developer Surveys, would you consider yourself less competent than average, of the same competency as average, or more competent than average?"

Re: Please be more careful when interpreting the Stack Overflow Developer Survey

#15

Post by asaph » Sat, Apr 13, 2019, 3:34 PM UTC

> any conclusions drawn by such studies must be taken with a not-insignificant sized grain of salt.

Please be more careful about drawing conclusions about the statistical significance of salt grain size. ;)

Re: Please be more careful when interpreting the Stack Overflow Developer Survey

#16

Post by balfirevic » Sat, Apr 13, 2019, 3:39 PM UTC

prepend wrote: post #11

shadowmint wrote: post #4

> you cannot generalize from a non-random sample So, honest question: If any survey of any size can be ignored on the basis that the sample is not random, then how is any survey meaningful? Isn’t this a self defeating argue? You can’t prove the sample is random, all you can do is show differences between samples and suggest its not consistent ... but how do we go away and prove that some other survey we’re comparing…

The key is in how you randomly select the sample from the population. This was the author’s point. Just because you have 90k SO respondents doesn’t mean you can say anything about developers as a population. You can say lots of stuff about SO users. Or maybe developers who use SO. But just because you have lots of responses doesn’t mean you know what developers or jugglers or farmers or whatever population interests…

It's not even a random sample of SO visitors, as the there is, at the very least, self-selection bias.

Re: Please be more careful when interpreting the Stack Overflow Developer Survey

#17

Post by balfirevic » Sat, Apr 13, 2019, 3:48 PM UTC

shadowmint wrote: post #4

> you cannot generalize from a non-random sample So, honest question: If any survey of any size can be ignored on the basis that the sample is not random, then how is any survey meaningful? Isn’t this a self defeating argue? You can’t prove the sample is random, all you can do is show differences between samples and suggest its not consistent ... but how do we go away and prove that some other survey we’re comparing…

> Am I missing something here? Everyone seems thoughorly convinced that this is perfectly normal.

What you're missing is that one of the intuitions you have is simply wrong. The intuition is that sample size can undo the ill effects of non-random sample. As stated in the original article and elsewhere in the comments, it cannot:

> It is an error to use the sample size of a non-random sample to support the underlying comparison with the population of interest. Sample size can decrease random error, but not bias

Re: Please be more careful when interpreting the Stack Overflow Developer Survey

#18

Post by leshow » Sat, Apr 13, 2019, 3:57 PM UTC

_pastel wrote: post #14

Next year, ask: "Compared to developers of similar position, experience, and willingness to respond to Stack Overflow Developer Surveys, would you consider yourself less competent than average, of the same competency as average, or more competent than average?"

How can you compare another developers willingness to respond to SO surveys to your own?

Re: Please be more careful when interpreting the Stack Overflow Developer Survey

#19

Post by dahart » Sat, Apr 13, 2019, 4:34 PM UTC

shadowmint wrote: post #4

> you cannot generalize from a non-random sample So, honest question: If any survey of any size can be ignored on the basis that the sample is not random, then how is any survey meaningful? Isn’t this a self defeating argue? You can’t prove the sample is random, all you can do is show differences between samples and suggest its not consistent ... but how do we go away and prove that some other survey we’re comparing…

> I don’t accept you can survey 90000 developers and cannot offer any generalization from those results without quantitatively proving there is an overwhelming sample bias

I didn’t see anyone point this out here yet specifically, but what you’re missing is that these 90k devs chose to respond to the survey, and the group is made of only SO participants, they were not developers selected at random. That’s the problem here.

There is an overwhelming bias, and it has been proved. Stack Overflow admits that openly and Julia talked about it in her answer to the OP’s commentary:

“Developers from underrepresented groups in tech participate on Stack Overflow at lower rates, so we undersample those groups, compared to their participation in the software developer workforce. We have data that confirms that”

Re: Please be more careful when interpreting the Stack Overflow Developer Survey

#20

Post by dahart » Sat, Apr 13, 2019, 4:39 PM UTC

> We asked respondents to evaluate their own competence, for the specific work they do and years of experience they have, and almost 70% of respondents say they are above average while less than 10% think they are below average. This is statistically unlikely with a sample of over 70,000 developers who answered this question, to put it mildly.

I’m seeing a lot of argument about statistics, and very little about the wording of this question.

It seems to me that by asking specifically for competence “for the specific work you do” means that everyone gets to define who they’re comparing against.

What this means is that it’s entirely statistically possible for 70% of people to be above average relative to whoever they choose their peers to be. I might compare myself to you, and you don’t compare yourself to me, so we can both be above average.

In order to argue about the math and statistics, the question needs to be well defined, and the group to which we’re talking about averages needs to be the same for everyone. This question, the way its worded, practically guarantees that the group someone compares themselves to is different for every single respondent.