Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

One Year as a Data Scientist at Stack Overflow

varianceexplained.org

Post reply on HN
Print view View original

31–40 of 46 posts

Re: One Year as a Data Scientist at Stack Overflow

#31

Post by lovestats3 »

I just read your R course lessons and I find it very well explained, I enjoyed the lessons about data.table and ggplot2.

The beta distribution: In the free book "think stats Probability and Statistics for Programmers" there is a chapter about how the beta distribution can be used as a prior to model an unknown probability and how Bayes' Theorem allow us to update that prior with a posterior distribution that is also a Beta distribution, that important property is called the self-prior property of the beta. Since the two parameters of the beta in that intuitive explanation are just the number of experiments (battings) and the number of successes (runs) that example constitutes a very intuitive and clear way to explain what is Bayesian Statistics. I think that you would enjoy the think stats book, it is aimed for programmers and it tries hard to enhance intuition.

I also enjoyed how you describe the atmosphere in your office, it seems that you work in a lovely place in which statistics is a well respected tool and people try to explore and innovate in a fun way without excessive pain. Nice post, I enjoyed it.

Re: One Year as a Data Scientist at Stack Overflow

#32

Post by Declanomous »

mswen wrote: post #22
IanCal wrote: post #13

Earlier quoted context omitted.

Totally realistic. It'll depend on what the company is, and what they're looking for (and hopefully, this will match what they should be looking for). While people can often focus on applying the latest deep learning thought-vector approach to their BIG DATA, there's an enormous gulf between the common condition of data and this aspiration. You don't need PhD level stats and machine learning to apply the things that…

IanCal - I upvoted you but let me comment for extra emphasis. I remember coming out of PhD level quantitative social science studies in academia back in the late 90s where I was using K-means clustering, Factor analysis, multiple linear regression, ANOVA and more. When I moved into marketing research and dealing with company data it was shocking and disheartening how little of those skills I could actually deploy. Da…

I went into marketing from a biology background, and I felt the transition was very easy. Collecting data in biology, especially on an ecosystem scale, is extremely difficult, and your data has a lot of noise in it. How the data is collected and processed is almost more important than how the data was analyzed.

Collecting and processing data is going to be extremely time consuming and expensive regardless of what you do, but you can make it easier on yourself through smart experimental design. P-values are still widely used for analysis in biology, to the chagrin of most statisticians. I don't think it is as much of a problem as it comes across as though. Bayesian reasoning is inherently included in the scientific process as part of the experimental design stage. The reliance on p-values becomes a problem when journalists report on research findings in a single paper that is "significant" because of p-values. The professionals in the field are capable of balanced analysis of p-value based research, but we end up with issues like anti-vaxxers when research is presented outside the field of professionals.

Of course there are still ways the process can allow incorrect research to present itself. For one thing, journals are not interested in publishing studies that do not have "significant" results, which means that bad research can stick around for far too long.

I personally think that the solution to these problems is open data, since so much of the research depends on how the data was manipulated and 'cleaned' prior to analysis.

Re: One Year as a Data Scientist at Stack Overflow

#33

Post by dragonwriter »

amelius wrote: post #16
harshnisar wrote: post #15

Earlier quoted context omitted.

Interesting, how is optimising advertisement CTR a zero-sum game? Just curious.

From wikipedia [1]: > In game theory and economic theory, a zero-sum game is a mathematical representation of a situation in which each participant's gain (or loss) of utility is exactly balanced by the losses (or gains) of the utility of the other participant(s). Advertisement is (to first approximation) zero-sum in the sense that what you sell, your competitor will not sell. The contribution to society of the adver…

> Advertisement is (to first approximation) zero-sum in the sense that what you sell, your competitor will not sell.

No, its not. A major portion of the effort in advertising of most products goes into spreading knowledge and/or perceived need for the product category, not just competing for share of the existing demand.

> The contribution to society of the advertisement is zero.

Even in the purely competitive case, that's only clearly the case in the case where the products are perfect subsitutes with neither cost nor utility differences for the purchaser.

Re: One Year as a Data Scientist at Stack Overflow

#34

Post by lovestats3 »

lovestats3 wrote: post #31

I just read your R course lessons and I find it very well explained, I enjoyed the lessons about data.table and ggplot2. The beta distribution: In the free book "think stats Probability and Statistics for Programmers" there is a chapter about how the beta distribution can be used as a prior to model an unknown probability and how Bayes' Theorem allow us to update that prior with a posterior distribution that is also…

If I were to work in your place to try to match people to jobs I should study some psychology and NLP trying to predict from what an user write and answers what is his mental state and how that mental state fits in the jobs.

Unfortunately hiring seems to be a very difficult problem and I can imagine that many key features are hidden and can't be obtained since the candidate can't be put into a controlled experiment that should allow us to obtain deep information that is usually hidden, perhaps a way to gain more information and wealth is to communicate with your users and clients in such a way that what is now hidden can be measured and new features can be obtained. That is you need to think about a model for your users, and that model must use features related to mental and human capabilities.

Re: One Year as a Data Scientist at Stack Overflow

#35

Post by lovestats3 »

amelius wrote: post #10
marmaduke wrote: post #6

Ah man, I wanted that job. I guess I didn't waste enough time on the Internet answering questions during my PhD.

The article is a very nice write-up, but in the end he is still optimizing advertisement click-through rates (a zero-sum game). While interesting from a technical point of view, why not make yourself 1000x more useful to society by working, for example, on "cognitive health" problems? These are problems that lean heavily on statistics, and are interesting and imho more rewarding at the same time.

How is matching people to jobs a zero-sum game? If you are able to devise a method to find the ideal job for people I think that society will benefit from it, better products and services will be constructed, so this is in no way a zero-sum game. Is a win-win situation is jobs seekers and jobs providers are able to find the match the right candidate with the right job what require to pay the right price.

Re: One Year as a Data Scientist at Stack Overflow

#36

Post by nonbel »

Declanomous wrote: post #29
mkagenius wrote: post #24

> It makes me sad when brilliant software engineers open up Excel to make a line graph! Why do people get religious about tech.. funny, let him/her use excel for god sake its a great tool :)

Excel is a great tool, but its graphing ability leaves a lot to be desired. I feel that graphing in Excel has gotten worse as they have tried making it easier to use as well. I'm 99% sure the graphing engine in Office 2007-2016 is the same one as in 2003 and earlier, but all of the damn menus they have added slow down the graph making process so much. If you are trying to use Excel to make a graph for a reasonably te…

>"I graduated from college with a degree in biology, and I spent hundreds or thousands of hours in Excel manipulating data and graphing...creating a graph with 254 series will probably take the better part of a week."

I don't see how this is an argument for using excel. You could have spent a small fraction of that time learning the basics of R and gotten everything afterwards done 10-100x faster. I know, because I used to be one of the people who spent insane amount of time on incredibly basic stuff. That said I still do sometimes use excel to inspect data and for some other simple tasks.

Re: One Year as a Data Scientist at Stack Overflow

#37

Post by var_explained »

amelius wrote: post #12

> For that, I might look at another source of data, Stack Overflow Careers profiles, and see which technologies tend to be used by the same developers > http://varianceexplained.org/images/network2.jpeg This shows "git" and "github" in a separate cluster from "C++" and "Python"? I don't understand this. These tools are used regardless of what other technologies are being used. For example, there are many Python and m…

mikk14 is right; the connections were filtered out below a particular threshold (set that threshold too low and everything is connected to everything; set it too high and you miss meaningful connections).

But note that precisely because git/github is used in combination with almost all other tags, it doesn't have a high correlation with any particular technology. A correlation (roughly) means "If I know you use tag X, you're more likely to also use tag Y." But knowing someone uses git doesn't let you guess what other technologies you use, because as you note they can use almost anything.

In short, if you're connected to everything, that means you're correlated with nothing.

Re: One Year as a Data Scientist at Stack Overflow

#38

Post by kafkaesq »

mkagenius wrote: post #24

> It makes me sad when brilliant software engineers open up Excel to make a line graph! Why do people get religious about tech.. funny, let him/her use excel for god sake its a great tool :)

Nothing "religious" about that observation; quite the opposite, in fact.

In particular: it's not so much whether the tool "works" or not, but the side effects (how well it plays with other tools; and hence, the extent to which it can be integrated into larger, maintainable projects) that need to taken into consideration, also. Which is why better engineers detect more than a whiff of bad "design smell" when they hear someone say "just use Excel" (most of the time).

Re: One Year as a Data Scientist at Stack Overflow

#39

Post by var_explained »

amelius wrote: post #20
marketforlemmas wrote: post #17

Earlier quoted context omitted.

I don't think advertising for jobs is a zero-sum game, especially if the matching algorithm is good enough to match employers and employees that had no knowledge of each other before. If you are able to reduce those search frictions, you have created value. Several economic professors won the Nobel prize for their work in this area: http://economix.blogs.nytimes.com/2010/10/11/the-work-behind... https://en.wikipedia.…

Well, those Nobel prizes are for a specific theory. In practice, in this case, you see developers looking to get problems solved (or solving problems), and getting distracted to look for another job. So even if you are removing market friction, there's a huge cost in people switching (or even getting distracted all the time). In my opinion, if people are looking for a job, they should go to a job-hunting website (eve…

There are many developers looking for a good job. There are many companies having difficulty finding qualified developers.

Our goal is to solve those problems, and it is not a zero-sum game.

Re: One Year as a Data Scientist at Stack Overflow

#40

Post by Declanomous »

nonbel wrote: post #36
Declanomous wrote: post #29

Earlier quoted context omitted.

Excel is a great tool, but its graphing ability leaves a lot to be desired. I feel that graphing in Excel has gotten worse as they have tried making it easier to use as well. I'm 99% sure the graphing engine in Office 2007-2016 is the same one as in 2003 and earlier, but all of the damn menus they have added slow down the graph making process so much. If you are trying to use Excel to make a graph for a reasonably te…

>"I graduated from college with a degree in biology, and I spent hundreds or thousands of hours in Excel manipulating data and graphing...creating a graph with 254 series will probably take the better part of a week." I don't see how this is an argument for using excel. You could have spent a small fraction of that time learning the basics of R and gotten everything afterwards done 10-100x faster. I know, because I u…

I wasn't arguing that one should use Excel. I wasn't arguing that you shouldn't use Excel either. I was arguing the exact same point you just made - you should use the appropriate tool for the job.

Excel is a great tool for manipulating data, especially if you are working with other people. Excel produces serviceable graphs, but if you need to produce a graph of any sort of complexity (and know how to program) R is certainly a better tool. Sometimes you need a quick visualization of some very simple data. I'd prefer to use Excel rather than R or Python at that point.

Post reply on HN