Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What's the state of the job market in data science and machine learning?

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 140 posts

Re: Ask HN: What's the state of the job market in data science and machine learning?

#21

Post by wjn0 »

whenwillitstop wrote: post #4

I am curious about this as well. I think the difference between machine learning and software engineering is that companies may only need a few dozen machine learning engineers. They may need thousands of software engineers. There may be increasing demand, but the demand will never reach the demand of software engineering. Except at the premium ultra competitive level, where a data scientist who is globally known can…

It seems that this is true for now (for 'traditional companies').

Soon, however, one could argue that 'traditional companies' will no longer be the norm - data science, ML, etc. will play such a crucial role in the majority of tech firms that the number of companies using it will rise. That's when I expect we'll see a huge portion of software engineers knowing ML concepts. Alternatively, I wonder if we might see the rise of smaller companies contracting out all of their ML to larger ones.

I would also be curious to know if ML background helps one to get a job at a place like Amazon/Google for even 'traditional' positions right now. The amount of data they have now must drive demand for engineers who can write software that takes advantage of it, regardless of position. Of course, like you said, they'll always require engineers to fill more traditional roles with no data interaction.

Re: Ask HN: What's the state of the job market in data science and machine learning?

#22

Post by DrNuke »

hardtke wrote: post #18
runT1ME wrote: post #13

Earlier quoted context omitted.

> The key difference is whether the candidate truly understands the mathematical and statistical basis of machine learning Can you elaborate on this, and at what level? Are you talking about a PhD level of understanding of cutting edge mathematics, or do you mean understand the basics, or somewhere in between?

Somewhere in between. A simple question I use is "how do I decide whether to add a feature to a classification model?" Most candidate are fine until I bring up the topic of correlated features.

Are you hiring for finance? In Europe it still depends very much on the industry: some mean coding, some maths, some infrastructure, all want data preparation and in-depth stats. edit: other than understanding the business fundamentals, of course.

Re: Ask HN: What's the state of the job market in data science and machine learning?

#23

Post by rm999 »

Speaking for NYC, but I imagine silicon valley is similar.

The supply-demand dynamics have changed a lot in the last couple years. I'd roughly break it out into two groups: people with work experience + strong software development skills, and those without. The first group is in higher demand than ever, and tend to add a lot of value to companies that really need it.

The second group has gotten extremely crowded, especially from STEM graduates - usually with a masters or phd - who have completed MOOCs or bootcamps. Supply keeps growing while demand is flat or shrinking (especially as executives get burned by "data scientists" who don't know how to help them build things of value). There's a huge crunch here; a lot of people I know in this group have been searching for jobs for months, eventually settling for a low quality job or giving up entirely :(

Re: Ask HN: What's the state of the job market in data science and machine learning?

#24

Post by akvadrako »

hardtke wrote: post #18
runT1ME wrote: post #13

Earlier quoted context omitted.

> The key difference is whether the candidate truly understands the mathematical and statistical basis of machine learning Can you elaborate on this, and at what level? Are you talking about a PhD level of understanding of cutting edge mathematics, or do you mean understand the basics, or somewhere in between?

Somewhere in between. A simple question I use is "how do I decide whether to add a feature to a classification model?" Most candidate are fine until I bring up the topic of correlated features.

Do you mean like non-orthogonal dimensions?

Re: Ask HN: What's the state of the job market in data science and machine learning?

#26

Post by TYPE_FASTER »

In my limited experience, there's a difference between a data scientist who can process data given data and a set of questions about it, and a data scientist who can figure out what data you need, and the questions that need to be answered.

I think making the transition from the first role to the second role comes with experience, both with the toolsets, and thinking about the problem as a whole.

Re: Ask HN: What's the state of the job market in data science and machine learning?

#27

Post by photoJ »

hardtke wrote: post #18
runT1ME wrote: post #13

Earlier quoted context omitted.

> The key difference is whether the candidate truly understands the mathematical and statistical basis of machine learning Can you elaborate on this, and at what level? Are you talking about a PhD level of understanding of cutting edge mathematics, or do you mean understand the basics, or somewhere in between?

Somewhere in between. A simple question I use is "how do I decide whether to add a feature to a classification model?" Most candidate are fine until I bring up the topic of correlated features.

That's terrible.

Re: Ask HN: What's the state of the job market in data science and machine learning?

#28

Post by TXV »

rm999 wrote: post #23

Speaking for NYC, but I imagine silicon valley is similar. The supply-demand dynamics have changed a lot in the last couple years. I'd roughly break it out into two groups: people with work experience + strong software development skills, and those without. The first group is in higher demand than ever, and tend to add a lot of value to companies that really need it. The second group has gotten extremely crowded, esp…

I think what you say is easily applicable to software engineering in general. Data science maybe is a field that is even more negatively impacted by bad hires because the threshold after which you start adding value to the company is higher.

Re: Ask HN: What's the state of the job market in data science and machine learning?

#29

Post by pcsanwald »

rm999 wrote: post #23

Speaking for NYC, but I imagine silicon valley is similar. The supply-demand dynamics have changed a lot in the last couple years. I'd roughly break it out into two groups: people with work experience + strong software development skills, and those without. The first group is in higher demand than ever, and tend to add a lot of value to companies that really need it. The second group has gotten extremely crowded, esp…

I've only been hiring DS folks since 2012, but my experience matches what you've said exactly. The biggest differentiator I've seen is to be able to participate in actually building production quality systems vs being proficient enough in R or python to hack together a prototype on a very small dataset.

The former kind of data scientists were very successful at our company, the latter, not so much. Both categories I described usually had a STEM type PhD.

Re: Ask HN: What's the state of the job market in data science and machine learning?

#30

Post by shas3 »

runT1ME wrote: post #13
hardtke wrote: post #11

I hire machine learning engineers and data scientists. In my opinion there is a great shortage of truly qualified machine learning engineers. A lot of people are entering the market with a general knowledge of machine learning tools. These people should be considered analysts or product data scientists. When it comes to people that can build machine learning systems that work at scale, they are very rarely available…

> The key difference is whether the candidate truly understands the mathematical and statistical basis of machine learning Can you elaborate on this, and at what level? Are you talking about a PhD level of understanding of cutting edge mathematics, or do you mean understand the basics, or somewhere in between?

I think the following interview questions can help screen candidates (not an exhaustive list):

1. Asking a candidate to cast a non-standard problem of classification or estimation into a tractable optimization problem. (this is a very valuable skill that someone who has done good studies in numerical linear algebra/machine learning/stat/information theory/control systems/signal processing/math/etc. should be able to do)

2. Asking them to take an algorithm they have used and explain every step in deriving the algorithm. (It will help interviewer calibrate the level of learning in the interviewee. Also helps screen for indisciplined black-box users.)

3. Presenting challenging machine learning scenarios: using customized ensemble learning approaches, imbalanced data sets, noisy labels, multiple instances, different error metrics, etc. and seeing how interviewee approaches the problem from first principles (real world problems almost always involve some of these issues)

4. Testing their intuitions in "feature-engineering" for different types of data. (with the partial exception of cases where rigorous research/successful products show the utility of deep learning, one has to almost necessarily do a fair bit of feature-engineering)

Post reply on HN