Live data from Hacker News

Ask HN: Is the industry saturated with data scientists already?

news.ycombinator.com

21–30 of 38 posts

Re: Ask HN: Is the industry saturated with data scientists already?

#21

To be blunt, the market is saturated with people who call themselves “data scientists” but are actually just reasonably skilled software engineers with at best a college sophomore level understanding of math/stats. On the other hand, the market is nowhere near saturation for people with both advanced software engineering and math/stats skills (i.e. PhD-level).

I don't think that is correct. I don't think the market actually needs phd level stats skills. Its just like the market doesn't need phd level pure math or doesn't need phd level CS. The number of situations where these specialized knowledge is useful in industry is vanishingly small. Having a phd is more about being even considered for an interview for such a position. The actual day to day knowledge used will consist of topics that even a person with a bachelor could require without putting in years.

Re: Ask HN: Is the industry saturated with data scientists already?

#22

To be blunt, the market is saturated with people who call themselves “data scientists” but are actually just reasonably skilled software engineers with at best a college sophomore level understanding of math/stats. On the other hand, the market is nowhere near saturation for people with both advanced software engineering and math/stats skills (i.e. PhD-level).

I don't think that is correct. I don't think the market actually needs phd level stats skills. Its just like the market doesn't need phd level pure math or doesn't need phd level CS. The number of situations where these specialized knowledge is useful in industry is vanishingly small. Having a phd is more about being even considered for an interview for such a position. The actual day to day knowledge used will consi…

They are saying that the thing you are talking about right now is saturated - but positions that authentically need that level of expertise are not.

Re: Ask HN: Is the industry saturated with data scientists already?

#24

Earlier quoted context omitted.

I don't think that is correct. I don't think the market actually needs phd level stats skills. Its just like the market doesn't need phd level pure math or doesn't need phd level CS. The number of situations where these specialized knowledge is useful in industry is vanishingly small. Having a phd is more about being even considered for an interview for such a position. The actual day to day knowledge used will consi…

They are saying that the thing you are talking about right now is saturated - but positions that authentically need that level of expertise are not.

Thanks for making my point more clearly than I did!

As an aside, regarding whether people actually do PhD-level math at these jobs, the answer is indeed often no. However, the jobs can still require PhD-level experience. This is because for the average person, there is a lag between being able to merely learn concepts at a given level versus being able to actually synthesize those concepts to solve novel problems. It is relatively easy to learn a subject and solve exercises in that subject that you know pertain to the concepts you just studied, as you would encounter in a course. It is much harder to be given a problem out of the blue and realize what concepts are required to solve it, as you would encounter in a scientific career.

As a concrete example, I once was explaining neural networks to a bright college freshman. I showed him the forward pass equation, then asked him how he would optimize the network weights given said equation. Even though he learned the chain rule in his courses, he didn’t think to apply it to derive the backpropagation step. By contrast, a talented junior or senior can easily figure this out.

In my experience, for the average person, the learning/synthesis gap is usually a few years. Hence, your average new PhD-level data scientist would be capable of synthesizing advanced undergraduate material towards solving novel problems in their job. And there are a hell of a lot of data science jobs that require that.

Re: Ask HN: Is the industry saturated with data scientists already?

#25
I don't know, and it's really impossible to tell since things are changing quickly. People work as "data scientists", but there are headwinds and tailwinds. the main headwind, is that companies are cutting budgets due to the recession and dropping analysis groups that are part of cost centers. It's also easy to get the basic skills (coding/stats) done in your undergrad or masters w/o having research experience. The tailwinds, are that ML capabilities are improving day by day, so the potential to use that to make money are increasing. There's also a huge digital transformation happening, and companies have more data than ever before and potential to leverage that into savings, additional revenue, or new services.

When I started on my data science path, about 10 years ago, and there was no training pipeline, so when I dropped out of a PhD a few years later it wasn't that hard to get a data science job with the intersection of skills: math/stats/coding/research. Today that role is probably filled by someone graduating from an undergraduate or grad program, but I know the same company is still hiring for improvements on the research project I helped start.

Good data science, for me, is when you "apply predictive models to end user problems and ship solutions in products", but when I looked around for other jobs I realized that so few companies are able to act cross functionally to exploit the value of ML in products and services. Sure, finance does it, ads does it too, but it seems like the jobs I had access to were some ill-thought out skunkworks that a VP or exec thought was a good idea, or doing work tucked away in some business unit. There are like 10 individual problems there for YC to solve, but the more fundamental issue is that as long as we are still in the hype phase of data science, there will be incentive for business leaders to spend money on it in wasteful ways (at least for your career).

If you want to do data science or ML, it'd encourage you to find tech first companies that are actually using ML to solve real world problems for people, and avoid working on projects that haven't shipped. Also, stay under engineering orgs. In business units, you'll have a boss that doesn't understand what you do, and you'll be promoted out of tech.

Ultimately, I left data science and am now on an infrastructure team at a database company, which is just a better fit for values. If you can get into big tech or any tech first company, the data science is mostly figured out, but in my experience lots of companies aren't offering constructive experience. Good luck.

Re: Ask HN: Is the industry saturated with data scientists already?

#26
post #14

a side, but a sincere question: what do Data Scientists do and what can I expect one to produce as productive output? I've worked as an SDE on data engineering projects myself (Spark / Hadoop stuff) and have friends who are ML researches and develop things like better recommendation results. Never met a data scientist.

It's pretty nebulous, but the work output of a data scientist would be a predictive model, where those results are either useful for some business unit (forecasting), or capable of being shipped as part of a software system and product.

Using Uber as an example, a data scientist would figure out the algorithm/model for assigning the next driver when you request a ride, the model for the shortest route (maybe), and delivering a model to correct GPS works in cities with huge buildings so drivers know exactly where to pick people up when they stand next to skyscrapers.

It requires a ton of infrastructure to do good data science work, since you not only need to validate that the model works using the exact same data in testing/production, but you need to take code that a non-SWE runs, integrate it into the build, figure out the right operational metrics, then deploy as part of some release strategy.

The model is really the smallest part of that process, but occasionally, you can get a huge lift by having someone apply a lot of interesting math.

Re: Ask HN: Is the industry saturated with data scientists already?

#27
The problem with the Data Scientist market is that Data Scientist is a fuzzy definition. It's pretty clear what skillset makes a good software engineer, but nearly anyone can call themselves a Data Scientist, even if they just clicking around in Excel. This has resulted in these bootcamps and certifications that "make you a Data Scientist in 14 days" - now everyone with minimal qualifications can apply for Data Scientist roles. That's why the market appears so crowded, and it is. People thought it's an easy and quick way to a high paying job, resulting in a flood of low-quality applicants for these positions.

There is still a lot of room for people who have strong engineering AND data engineering/ML/math/statistics skills. But then don't call yourself Data Scientist because that puts you into the same low-barrier camp as all the others. From my own experience it's a clear resume red flag: Almost anyone that market themselves as primarily a "Data Scientist" has little technical skills.

Re: Ask HN: Is the industry saturated with data scientists already?

#28
I believe, unless the role is understood at the top and given some slack to do properly, data scientists are setup to fail.

The expectations are generally wildly unrealistic and the work may touch on many departments. It is a minefield politically, unless it is a very clear priority for the company.

If the value is understood at the top data science can provide immense value.

When the media mentions x job being sexy or a shortage of x workers there is usually an agenda. I would not take those assertions at face value.

Re: Ask HN: Is the industry saturated with data scientists already?

#29
post #5

I think it's less a case of saturation and more a case of companies realizing that most data scientists don't actually deliver value commensurate to the salaries they were asking. Most companies simply don't have enough data, or don't have hygienic enough data, or don't have the engineering heft to build a reliable data pipeline, so the data scientists often find themselves set up to fail. I've seen that happen a few…

Well, where I work at, I was hired as a data scientist and end up doing everything related to data lol, engineering, analytics and what not. It doesn't matter as long as they pay me to do that.

Same here!

I ended up shifting more towards the dev side - devops, software engineering, etc.

Re: Ask HN: Is the industry saturated with data scientists already?

#30
post #5

I think it's less a case of saturation and more a case of companies realizing that most data scientists don't actually deliver value commensurate to the salaries they were asking. Most companies simply don't have enough data, or don't have hygienic enough data, or don't have the engineering heft to build a reliable data pipeline, so the data scientists often find themselves set up to fail. I've seen that happen a few…

In my experience it is common to meet managers who think they have far more data than is really the case. What data they do have tends to be inconsistent due to changes in processes and program changes. This is particularly gruesome when a migration between ERP systems had been undertaken or after a company merger or even reorganization. It is impossible for a data scientist to reliably identify the data problems of…

> In my experience it is common to meet managers who think they have far more data than is really the case. What data they do have tends to be inconsistent…

Can also back this up 100%.

Too many times I’ve had “we have some x data, we need you to do y”. Turns out, the data they “have” is a 15 line excel file of unknown provenance and replicability; nobody in the dev/marketing/finance/etc team has the time, interest, or project alignment to care what you want, and what they’re trying to do needs a team of phd’s anyways, but all of this falls on deaf ears.

Post reply on HN