Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What's the state of the job market in data science and machine learning?

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 140 posts

Re: Ask HN: What's the state of the job market in data science and machine learning?

#31

Post by user5994461 »

pcsanwald wrote: post #29
rm999 wrote: post #23

Speaking for NYC, but I imagine silicon valley is similar. The supply-demand dynamics have changed a lot in the last couple years. I'd roughly break it out into two groups: people with work experience + strong software development skills, and those without. The first group is in higher demand than ever, and tend to add a lot of value to companies that really need it. The second group has gotten extremely crowded, esp…

I've only been hiring DS folks since 2012, but my experience matches what you've said exactly. The biggest differentiator I've seen is to be able to participate in actually building production quality systems vs being proficient enough in R or python to hack together a prototype on a very small dataset. The former kind of data scientists were very successful at our company, the latter, not so much. Both categories I…

That sounds weird to me. Does American PhD don't have to work for a few years at real companies as part of the PhD curriculum?

Re: Ask HN: What's the state of the job market in data science and machine learning?

#32

Post by ylem »

user5994461 wrote: post #31
pcsanwald wrote: post #29

Earlier quoted context omitted.

I've only been hiring DS folks since 2012, but my experience matches what you've said exactly. The biggest differentiator I've seen is to be able to participate in actually building production quality systems vs being proficient enough in R or python to hack together a prototype on a very small dataset. The former kind of data scientists were very successful at our company, the latter, not so much. Both categories I…

That sounds weird to me. Does American PhD don't have to work for a few years at real companies as part of the PhD curriculum?

I think the scenario they are describing is for say math/physics PhDs who are transitioning to data science. During their degrees, they concentrate on research, so they don't (on average) have real software engineering experience.

Re: Ask HN: What's the state of the job market in data science and machine learning?

#33

Post by wjn0 »

curiousgal wrote: post #16
wjn0 wrote: post #14

I'm an undergrad at a big university known for CS in Canada. The CS program here has several possible 'focuses'; 4 of 9 are related to ML/AI directly (computer vision, NLP, AI, scientific computing). 2 others require AI/ML/NN courses. The bias might stem from the fact that we have some huge names in AI doing research here, but the data points seem clear (we say undergraduate education is slow to catch on, right?): th…

>or simply because we can abstract a lot of this away (ex: TensorFlow). That would work up to the point a better abstraction tool/framework comes along. I'd never try to build a career on a single framework, because frameworks come and go.

Building a career around a framework is never a good idea. If you know your shit, it shouldn't matter what framework you're using.

Theano and TF, for example, both make similar abstractions: graphs and numerical functions on top of the same matrix library, even. I would suspect someone could move between the two fairly easily. The problem is that a programmer can use TF/Theano/etc.'s built-in gradient descent functions pulled from a tutorial with their data subbed in _instead_ of learning the details of backpropogation, end up with decent results, and claim to have a basic understanding of ML - when really, they've managed to avoid it almost completely.

Re: Ask HN: What's the state of the job market in data science and machine learning?

#34

Post by hellogoodbyeeee »

rm999 wrote: post #23

Speaking for NYC, but I imagine silicon valley is similar. The supply-demand dynamics have changed a lot in the last couple years. I'd roughly break it out into two groups: people with work experience + strong software development skills, and those without. The first group is in higher demand than ever, and tend to add a lot of value to companies that really need it. The second group has gotten extremely crowded, esp…

Do you have advice for someone in the second camp? I have a master's in econ and I'm comfortable in R but I don't have good development skills. I'm wondering if I need to give up on data science jobs for a bit and try and find an entry level software development job

Re: Ask HN: What's the state of the job market in data science and machine learning?

#35

Post by user5994461 »

wjn0 wrote: post #21
whenwillitstop wrote: post #4

I am curious about this as well. I think the difference between machine learning and software engineering is that companies may only need a few dozen machine learning engineers. They may need thousands of software engineers. There may be increasing demand, but the demand will never reach the demand of software engineering. Except at the premium ultra competitive level, where a data scientist who is globally known can…

It seems that this is true for now (for 'traditional companies'). Soon, however, one could argue that 'traditional companies' will no longer be the norm - data science, ML, etc. will play such a crucial role in the majority of tech firms that the number of companies using it will rise. That's when I expect we'll see a huge portion of software engineers knowing ML concepts. Alternatively, I wonder if we might see the…

You need a friend at GooMAzonSoft to refer you. It's always been and will always be the easiest way in. Then traditional uninteresting phone call and uninspired 6 hours on site with people who probably didn't read your resume.

Maybe if you have a good profile and you get lucky, you'll go interview straight for one group who's interested in you, but I wouldn't bet on that.

Re: Ask HN: What's the state of the job market in data science and machine learning?

#36

Post by shas3 »

rm999 wrote: post #23

Speaking for NYC, but I imagine silicon valley is similar. The supply-demand dynamics have changed a lot in the last couple years. I'd roughly break it out into two groups: people with work experience + strong software development skills, and those without. The first group is in higher demand than ever, and tend to add a lot of value to companies that really need it. The second group has gotten extremely crowded, esp…

Frankly, MOOC-type machine learning- being able to do plain vanilla logistic regression or black-box deep learning techniques is not enough to get a job. This knowledge and experience has to be paired with one or more strengths: excellence in programming, grad-level math/stat/numerical skills/theoretical machine learning, domain specific expertise or experience (e.g. vision, audio, natural language, networks, geophysics, biomed, finance, etc.), proven ability to learn and adapt very fast.

Re: Ask HN: What's the state of the job market in data science and machine learning?

#37

Post by user5994461 »

curiousgal wrote: post #16
wjn0 wrote: post #14

I'm an undergrad at a big university known for CS in Canada. The CS program here has several possible 'focuses'; 4 of 9 are related to ML/AI directly (computer vision, NLP, AI, scientific computing). 2 others require AI/ML/NN courses. The bias might stem from the fact that we have some huge names in AI doing research here, but the data points seem clear (we say undergraduate education is slow to catch on, right?): th…

>or simply because we can abstract a lot of this away (ex: TensorFlow). That would work up to the point a better abstraction tool/framework comes along. I'd never try to build a career on a single framework, because frameworks come and go.

Build your career on whatever tool(s) companies want.

There's only in web development where the hype change every year. (And even there there are plenty of companies that are lagging enough behind to still have opportunities in the old thing).

Re: Ask HN: What's the state of the job market in data science and machine learning?

#38

Post by rafinha »

rm999 wrote: post #23

Speaking for NYC, but I imagine silicon valley is similar. The supply-demand dynamics have changed a lot in the last couple years. I'd roughly break it out into two groups: people with work experience + strong software development skills, and those without. The first group is in higher demand than ever, and tend to add a lot of value to companies that really need it. The second group has gotten extremely crowded, esp…

Isn't it 2 profiles of engineers? Those who make production code and those who work in prototypes? One thing is to understand why your model isn't converging, another is how to scale it up...

Re: Ask HN: What's the state of the job market in data science and machine learning?

#39

Post by anton_tarasenko »

hellogoodbyeeee wrote: post #34
rm999 wrote: post #23

Speaking for NYC, but I imagine silicon valley is similar. The supply-demand dynamics have changed a lot in the last couple years. I'd roughly break it out into two groups: people with work experience + strong software development skills, and those without. The first group is in higher demand than ever, and tend to add a lot of value to companies that really need it. The second group has gotten extremely crowded, esp…

Do you have advice for someone in the second camp? I have a master's in econ and I'm comfortable in R but I don't have good development skills. I'm wondering if I need to give up on data science jobs for a bit and try and find an entry level software development job

Target research-intensive jobs. Job descriptions are clear about what sort of machine learning professionals the company is looking for. In most cases, they want data engineers and developers who can make big, scalable production. Not researchers.

Just find who's looking for economists, specifically. Consulting firms and government agencies hire economists, but they don't call them data scientists. Investment funds may be interested, though hedge funds prefer math and physics graduates because they are easier to train.

Re: Ask HN: What's the state of the job market in data science and machine learning?

#40

Post by audleman »

rafinha wrote: post #38
rm999 wrote: post #23

Speaking for NYC, but I imagine silicon valley is similar. The supply-demand dynamics have changed a lot in the last couple years. I'd roughly break it out into two groups: people with work experience + strong software development skills, and those without. The first group is in higher demand than ever, and tend to add a lot of value to companies that really need it. The second group has gotten extremely crowded, esp…

Isn't it 2 profiles of engineers? Those who make production code and those who work in prototypes? One thing is to understand why your model isn't converging, another is how to scale it up...

A machine learning model is core to my company, and core to that is pulling in, cleaning, and working with large dataset from a variety of platforms. This means:

- excellence in being able to clean data at the column level for millions of data points - knowing how to work with such large scale data in a time efficient manner. One of our newer hires worked at the Postgres/SSD level to optimize and got it to where he could produce a full set within 5 minutes. Before that it was once every few months.

Being able to do these things is a prerequisite to building even a prototype of a model, and it requires substantial real-world programming experience to deal with those complexities.

Post reply on HN