Live data from Hacker News

One Year as a Data Scientist at Stack Overflow

varianceexplained.org

11–20 of 46 posts

Re: One Year as a Data Scientist at Stack Overflow

#12
> For that, I might look at another source of data, Stack Overflow Careers profiles, and see which technologies tend to be used by the same developers

> http://varianceexplained.org/images/network2.jpeg

This shows "git" and "github" in a separate cluster from "C++" and "Python"? I don't understand this. These tools are used regardless of what other technologies are being used. For example, there are many Python and many C++ projects on github.

Re: One Year as a Data Scientist at Stack Overflow

#13

I always liked data crunching, databases and data topics in general. Also, most of the software development knowledge I have I didn't learn in the official school curriculum, but rather from books, online courses and real-world experience. Now, having that in mind, how realistic is for a guy in mid-30s, a very good software developer (enthusiastic about functional programming, if that matters) to pick up enough knowl…

Totally realistic. It'll depend on what the company is, and what they're looking for (and hopefully, this will match what they should be looking for).

While people can often focus on applying the latest deep learning thought-vector approach to their BIG DATA, there's an enormous gulf between the common condition of data and this aspiration.

You don't need PhD level stats and machine learning to apply the things that many companies actually could benefit from. Storing, maintaining and managing the data properly is a start. Then working with the people in the business to get insights from the data they have. Often simple aggregations and visualisations can provide enormous benefit. Being able to show correlations, and sometimes even just being able to show how noisy things are can be important.

Big questions: How is reality different from what we think? How are these differences important for the company?

That might be a correlation that we don't expect, or a lack of one we think is there. Part of a next step might be to become more "pro-active" and design new experiments to answer questions that can't quite be answered yet due to a lack of data. Beyond that you're heading towards bringing a new feature into a product.

Does your current company have some data? Do they spend a lot of time emailing spreadsheets around, or have salesforce or a proper database? See if you can make something useful for your work that's based on an analysis of that data (and possibly do some work to smooth those workflows of passing data around).

Re: One Year as a Data Scientist at Stack Overflow

#14

I always liked data crunching, databases and data topics in general. Also, most of the software development knowledge I have I didn't learn in the official school curriculum, but rather from books, online courses and real-world experience. Now, having that in mind, how realistic is for a guy in mid-30s, a very good software developer (enthusiastic about functional programming, if that matters) to pick up enough knowl…

To take a page from the authors playbook...create some artifacts. Find interesting data sets, do interesting stuff with said data sets and write about it :)

Re: One Year as a Data Scientist at Stack Overflow

#15
post #10

Ah man, I wanted that job. I guess I didn't waste enough time on the Internet answering questions during my PhD.

The article is a very nice write-up, but in the end he is still optimizing advertisement click-through rates (a zero-sum game). While interesting from a technical point of view, why not make yourself 1000x more useful to society by working, for example, on "cognitive health" problems? These are problems that lean heavily on statistics, and are interesting and imho more rewarding at the same time.

Interesting, how is optimising advertisement CTR a zero-sum game? Just curious.

Re: One Year as a Data Scientist at Stack Overflow

#16
post #10

Earlier quoted context omitted.

The article is a very nice write-up, but in the end he is still optimizing advertisement click-through rates (a zero-sum game). While interesting from a technical point of view, why not make yourself 1000x more useful to society by working, for example, on "cognitive health" problems? These are problems that lean heavily on statistics, and are interesting and imho more rewarding at the same time.

Interesting, how is optimising advertisement CTR a zero-sum game? Just curious.

From wikipedia [1]:

> In game theory and economic theory, a zero-sum game is a mathematical representation of a situation in which each participant's gain (or loss) of utility is exactly balanced by the losses (or gains) of the utility of the other participant(s).

Advertisement is (to first approximation) zero-sum in the sense that what you sell, your competitor will not sell. The contribution to society of the advertisement is zero. You can say that you have provided the customer the service of making them aware of the product, but that is only in second approximation, as customers generally do not want this service. Also, they can only spend their money once. Further, ads cost money, so perhaps we can even call it a negative-sum game :)

[1] https://en.wikipedia.org/wiki/Zero-sum_game

Re: One Year as a Data Scientist at Stack Overflow

#17
post #16

Earlier quoted context omitted.

Interesting, how is optimising advertisement CTR a zero-sum game? Just curious.

From wikipedia [1]: > In game theory and economic theory, a zero-sum game is a mathematical representation of a situation in which each participant's gain (or loss) of utility is exactly balanced by the losses (or gains) of the utility of the other participant(s). Advertisement is (to first approximation) zero-sum in the sense that what you sell, your competitor will not sell. The contribution to society of the adver…

I don't think advertising for jobs is a zero-sum game, especially if the matching algorithm is good enough to match employers and employees that had no knowledge of each other before. If you are able to reduce those search frictions, you have created value. Several economic professors won the Nobel prize for their work in this area:

http://economix.blogs.nytimes.com/2010/10/11/the-work-behind...

https://en.wikipedia.org/wiki/Search_theory

Re: One Year as a Data Scientist at Stack Overflow

#18

I always liked data crunching, databases and data topics in general. Also, most of the software development knowledge I have I didn't learn in the official school curriculum, but rather from books, online courses and real-world experience. Now, having that in mind, how realistic is for a guy in mid-30s, a very good software developer (enthusiastic about functional programming, if that matters) to pick up enough knowl…

Absolutely realistic, though it helps to have good intuition about math. I've seen smart people without data science experience make the leap after a 12 week boot camp, or finishing the Johns Hopkins Coursera certificate. If you're good at picking stuff up online, the latter is probably the best start.

There is such a shortage of data scientists that most companies will settle for 2 out of 3 of the trinity of Stats, Programming and Domain Knowledge. If the candidate is smart enough, sometimes 1 out of 3 is enough.

Re: One Year as a Data Scientist at Stack Overflow

#19
post #12

> For that, I might look at another source of data, Stack Overflow Careers profiles, and see which technologies tend to be used by the same developers > http://varianceexplained.org/images/network2.jpeg This shows "git" and "github" in a separate cluster from "C++" and "Python"? I don't understand this. These tools are used regardless of what other technologies are being used. For example, there are many Python and m…

That is a heavily-filtered network: if he didn't drop most of the weak connections he'd end up with everything connected to everything else. On the other hand, if you don't use a sophisticated way to define what a "noisy" edge is, you'll end up with some curious cases like the one you point out. He might have used a naive global threshold -- it's the easiest way to go about it: drop all connections with weight lower than x. But it's also very wrong most of the times :-) Something like the disparity filter [1] works usually well, although you have sure that its null model hypothesis is aligned with what you think is the generative process of your network. The field is "network backboning" and it's a nice one from network science.

[1] https://en.wikipedia.org/wiki/Disparity_filter_algorithm_of_...

Re: One Year as a Data Scientist at Stack Overflow

#20
post #16

Earlier quoted context omitted.

From wikipedia [1]: > In game theory and economic theory, a zero-sum game is a mathematical representation of a situation in which each participant's gain (or loss) of utility is exactly balanced by the losses (or gains) of the utility of the other participant(s). Advertisement is (to first approximation) zero-sum in the sense that what you sell, your competitor will not sell. The contribution to society of the adver…

I don't think advertising for jobs is a zero-sum game, especially if the matching algorithm is good enough to match employers and employees that had no knowledge of each other before. If you are able to reduce those search frictions, you have created value. Several economic professors won the Nobel prize for their work in this area: http://economix.blogs.nytimes.com/2010/10/11/the-work-behind... https://en.wikipedia.…

Well, those Nobel prizes are for a specific theory. In practice, in this case, you see developers looking to get problems solved (or solving problems), and getting distracted to look for another job. So even if you are removing market friction, there's a huge cost in people switching (or even getting distracted all the time). In my opinion, if people are looking for a job, they should go to a job-hunting website (even if it has lower-quality data about them).

Anyway, it would be nice to have the effects properly quantified.

Post reply on HN