To be blunt, the market is saturated with people who call themselves “data scientists” but are actually just reasonably skilled software engineers with at best a college sophomore level understanding of math/stats. On the other hand, the market is nowhere near saturation for people with both advanced software engineering and math/stats skills (i.e. PhD-level).
Ask HN: Is the industry saturated with data scientists already?
21–30 of 38 posts
Re: Ask HN: Is the industry saturated with data scientists already?
#22To be blunt, the market is saturated with people who call themselves “data scientists” but are actually just reasonably skilled software engineers with at best a college sophomore level understanding of math/stats. On the other hand, the market is nowhere near saturation for people with both advanced software engineering and math/stats skills (i.e. PhD-level).
I don't think that is correct. I don't think the market actually needs phd level stats skills. Its just like the market doesn't need phd level pure math or doesn't need phd level CS. The number of situations where these specialized knowledge is useful in industry is vanishingly small. Having a phd is more about being even considered for an interview for such a position. The actual day to day knowledge used will consi…
Re: Ask HN: Is the industry saturated with data scientists already?
#23Re: Ask HN: Is the industry saturated with data scientists already?
#24Earlier quoted context omitted.
I don't think that is correct. I don't think the market actually needs phd level stats skills. Its just like the market doesn't need phd level pure math or doesn't need phd level CS. The number of situations where these specialized knowledge is useful in industry is vanishingly small. Having a phd is more about being even considered for an interview for such a position. The actual day to day knowledge used will consi…
They are saying that the thing you are talking about right now is saturated - but positions that authentically need that level of expertise are not.
As an aside, regarding whether people actually do PhD-level math at these jobs, the answer is indeed often no. However, the jobs can still require PhD-level experience. This is because for the average person, there is a lag between being able to merely learn concepts at a given level versus being able to actually synthesize those concepts to solve novel problems. It is relatively easy to learn a subject and solve exercises in that subject that you know pertain to the concepts you just studied, as you would encounter in a course. It is much harder to be given a problem out of the blue and realize what concepts are required to solve it, as you would encounter in a scientific career.
As a concrete example, I once was explaining neural networks to a bright college freshman. I showed him the forward pass equation, then asked him how he would optimize the network weights given said equation. Even though he learned the chain rule in his courses, he didn’t think to apply it to derive the backpropagation step. By contrast, a talented junior or senior can easily figure this out.
In my experience, for the average person, the learning/synthesis gap is usually a few years. Hence, your average new PhD-level data scientist would be capable of synthesizing advanced undergraduate material towards solving novel problems in their job. And there are a hell of a lot of data science jobs that require that.
Re: Ask HN: Is the industry saturated with data scientists already?
#25When I started on my data science path, about 10 years ago, and there was no training pipeline, so when I dropped out of a PhD a few years later it wasn't that hard to get a data science job with the intersection of skills: math/stats/coding/research. Today that role is probably filled by someone graduating from an undergraduate or grad program, but I know the same company is still hiring for improvements on the research project I helped start.
Good data science, for me, is when you "apply predictive models to end user problems and ship solutions in products", but when I looked around for other jobs I realized that so few companies are able to act cross functionally to exploit the value of ML in products and services. Sure, finance does it, ads does it too, but it seems like the jobs I had access to were some ill-thought out skunkworks that a VP or exec thought was a good idea, or doing work tucked away in some business unit. There are like 10 individual problems there for YC to solve, but the more fundamental issue is that as long as we are still in the hype phase of data science, there will be incentive for business leaders to spend money on it in wasteful ways (at least for your career).
If you want to do data science or ML, it'd encourage you to find tech first companies that are actually using ML to solve real world problems for people, and avoid working on projects that haven't shipped. Also, stay under engineering orgs. In business units, you'll have a boss that doesn't understand what you do, and you'll be promoted out of tech.
Ultimately, I left data science and am now on an infrastructure team at a database company, which is just a better fit for values. If you can get into big tech or any tech first company, the data science is mostly figured out, but in my experience lots of companies aren't offering constructive experience. Good luck.
Re: Ask HN: Is the industry saturated with data scientists already?
#26a side, but a sincere question: what do Data Scientists do and what can I expect one to produce as productive output? I've worked as an SDE on data engineering projects myself (Spark / Hadoop stuff) and have friends who are ML researches and develop things like better recommendation results. Never met a data scientist.
Using Uber as an example, a data scientist would figure out the algorithm/model for assigning the next driver when you request a ride, the model for the shortest route (maybe), and delivering a model to correct GPS works in cities with huge buildings so drivers know exactly where to pick people up when they stand next to skyscrapers.
It requires a ton of infrastructure to do good data science work, since you not only need to validate that the model works using the exact same data in testing/production, but you need to take code that a non-SWE runs, integrate it into the build, figure out the right operational metrics, then deploy as part of some release strategy.
The model is really the smallest part of that process, but occasionally, you can get a huge lift by having someone apply a lot of interesting math.
Re: Ask HN: Is the industry saturated with data scientists already?
#27There is still a lot of room for people who have strong engineering AND data engineering/ML/math/statistics skills. But then don't call yourself Data Scientist because that puts you into the same low-barrier camp as all the others. From my own experience it's a clear resume red flag: Almost anyone that market themselves as primarily a "Data Scientist" has little technical skills.
Re: Ask HN: Is the industry saturated with data scientists already?
#28The expectations are generally wildly unrealistic and the work may touch on many departments. It is a minefield politically, unless it is a very clear priority for the company.
If the value is understood at the top data science can provide immense value.
When the media mentions x job being sexy or a shortage of x workers there is usually an agenda. I would not take those assertions at face value.
Re: Ask HN: Is the industry saturated with data scientists already?
#29I think it's less a case of saturation and more a case of companies realizing that most data scientists don't actually deliver value commensurate to the salaries they were asking. Most companies simply don't have enough data, or don't have hygienic enough data, or don't have the engineering heft to build a reliable data pipeline, so the data scientists often find themselves set up to fail. I've seen that happen a few…
Well, where I work at, I was hired as a data scientist and end up doing everything related to data lol, engineering, analytics and what not. It doesn't matter as long as they pay me to do that.
I ended up shifting more towards the dev side - devops, software engineering, etc.
Re: Ask HN: Is the industry saturated with data scientists already?
#30I think it's less a case of saturation and more a case of companies realizing that most data scientists don't actually deliver value commensurate to the salaries they were asking. Most companies simply don't have enough data, or don't have hygienic enough data, or don't have the engineering heft to build a reliable data pipeline, so the data scientists often find themselves set up to fail. I've seen that happen a few…
In my experience it is common to meet managers who think they have far more data than is really the case. What data they do have tends to be inconsistent due to changes in processes and program changes. This is particularly gruesome when a migration between ERP systems had been undertaken or after a company merger or even reorganization. It is impossible for a data scientist to reliably identify the data problems of…
Can also back this up 100%.
Too many times I’ve had “we have some x data, we need you to do y”. Turns out, the data they “have” is a 15 line excel file of unknown provenance and replicability; nobody in the dev/marketing/finance/etc team has the time, interest, or project alignment to care what you want, and what they’re trying to do needs a team of phd’s anyways, but all of this falls on deaf ears.