Ask HN: What's the state of the job market in data science and machine learning? • Hacker News Board

Re: Ask HN: What's the state of the job market in data science and machine learning?

#101

Post by stared » Thu, Dec 22, 2016, 11:21 AM UTC

fnbr wrote: post #70

stared wrote: post #5

I have only anecdotal experience (I live in Warsaw, but do contracts mostly for Poland, UK and US). General data science is in need. I can get contracts easily, I know that people looking for competent people need to wait; especially as it is a skill much harder to pick than, say, front-end web dev (unless someone starts from a highly quantitive background like physics, modelling in biology, etc). My general impressi…

How do you find contracts? I'm interested in doing contract data science work, but I don't know how to start finding interested potential clients.

In the last ~1.5 years it is solely people contacting me. (But I give a lot of talks, workshops, and the chain of recommendations is going.)

One day I want to write how I get started, but I am not sure which steps were essential, which - irrelevant. And many things are not ones one can replicate.

Re: Ask HN: What's the state of the job market in data science and machine learning?

#102

Post by taway_1212 » Thu, Dec 22, 2016, 11:22 AM UTC

freyr wrote: post #43

platz wrote: post #2

I considered a graduate program in data science, but compared to average programmer salaries, it doesn't seem like data science pays all that much (excluding data science jobs for PHD's in silicon valley). It's more interesting that programming, but seems like a much tighter market with no discernible demand driving salaries up.

Interesting. I have a statistics/data science background, but personally I find programming much more satisfying. Programming is a tool to create and synthesize. It leads to new products, companies, and solutions. Data science is analysis, not synthesis. You collect data, you interpret it, you move on to other data. Nothing gets created, which for me, is a deal breaker for job satisfaction.

The premise is often that, as a programmer, you are a part of tightly controlled agile/SCRUM team, while DS get much more independence in their jobs.

Re: Ask HN: What's the state of the job market in data science and machine learning?

#103

Post by wakkaflokka » Thu, Dec 22, 2016, 2:12 PM UTC

rm999 wrote: post #23

Speaking for NYC, but I imagine silicon valley is similar. The supply-demand dynamics have changed a lot in the last couple years. I'd roughly break it out into two groups: people with work experience + strong software development skills, and those without. The first group is in higher demand than ever, and tend to add a lot of value to companies that really need it. The second group has gotten extremely crowded, esp…

I'm coming from a PhD in STEM where I did a lot of application of basic ML to my (neuroscience) research, and it took me a good while to get a data science position. But once I got the job, I have been inundated with interview requests, both from recruiters and from specific companies, as in multiple a week (granted most are blast-em-all style recruiters, that I'm sure anybody with any tech skills get). Maybe it's because I'm on the East Coast? I do feel like everybody in neuro is jumping on the bandwagon, and that generates a bit of "you don't belong here" feeling from the CS- or math- educated crowd. But over time I think this will all level-out.

I'm not coming from a CS background and don't purport to know absolutely all of the details of all the mathematics and theory behind many of the machine learning algorithms that I use. I try my best daily to expand my knowledge, understand the algorithms, and apply them appropriately. I would hope that any company who is looking for someone who has a PhD-in-CS-or-ML could weed someone with lesser knowledge out during the interview process.

With that being said, a couple lines of code using SciKit Learn and all the default parameters is enough to impress many non-tech companies that are looking for a way to use 'predictive' in their marketing materials. And they pay very well for it. I get the feeling that provokes the ire of people who think those types of basic implementations belong to the traditional label of 'data analyst'.

For what it's worth, I work with data sets that aren't quite large enough to justify anything more than Python, Pandas, SKLearn, Luigi pipeline, and PySpark. The vast majority of my time is spent cleaning the data and generating features, must less on the hyperparameter tuning, model training side itself.

Anyways, I think I'm rambling a bit.

I just want to say that I LOVE this job, whatever the label is, or whatever the hype surrounding the label is, and I hope it's around for a while before it's automated...

Re: Ask HN: What's the state of the job market in data science and machine learning?

#104

Post by ikding » Thu, Dec 22, 2016, 2:15 PM UTC

drew wrote: post #68

rm999 wrote: post #23

Speaking for NYC, but I imagine silicon valley is similar. The supply-demand dynamics have changed a lot in the last couple years. I'd roughly break it out into two groups: people with work experience + strong software development skills, and those without. The first group is in higher demand than ever, and tend to add a lot of value to companies that really need it. The second group has gotten extremely crowded, esp…

The problem is DS is really 2-3 different disciplines under one nebulous title. What you're describing is folks who are prototyping and productionizing models. That's definitely in short supply, but random STEM PhDs are in no way competitive for those roles unless they're coming from CS programs + have work experience in production engineering. But that's by no means all of the DS field. There are lots of DS jobs whe…

"Not all problems are solved by systems; lots of problems are solved by better understanding the problem and then letting other specialists build the right solution."

Agreed wholeheartedly. Reminds me of another quote from http://www.john-foreman.com/blog/surviving-data-science-at-t... :

""" You know what can keep up with a rapidly changing business?

Solid summary analysis of data. Especially when conducted by an analyst who's paying attention, can identify what's happening in the business, and can communicate their analysis in that chaotic context.

Boring, I know. But if you're a nomad living out of a yurt, you dig a hole, not a sewer system. """

Re: Ask HN: What's the state of the job market in data science and machine learning?

#105

Post by bear_child » Thu, Dec 22, 2016, 2:42 PM UTC

nl wrote: post #84

I'm in Australia. I'm hiring 6 people in a range of roles between "pure" data scientists to more data engineer/SWE roles. The exact mix depends on who we can get. The ability to find good people is the biggest constraint on the work we do. Our current team ranges from applied mathematicians (as in they are Math professors) to people with traditional SWE backgrounds. Basically we are a long long way from saturated.

What is your company? I am Australian, finishing a PhD in mathematics and looking for a job.

Re: Ask HN: What's the state of the job market in data science and machine learning?

#106

Post by shepardrtc » Thu, Dec 22, 2016, 2:48 PM UTC

user5994461 wrote: post #35

wjn0 wrote: post #21

Earlier quoted context omitted.

It seems that this is true for now (for 'traditional companies'). Soon, however, one could argue that 'traditional companies' will no longer be the norm - data science, ML, etc. will play such a crucial role in the majority of tech firms that the number of companies using it will rise. That's when I expect we'll see a huge portion of software engineers knowing ML concepts. Alternatively, I wonder if we might see the…

You need a friend at GooMAzonSoft to refer you. It's always been and will always be the easiest way in. Then traditional uninteresting phone call and uninspired 6 hours on site with people who probably didn't read your resume. Maybe if you have a good profile and you get lucky, you'll go interview straight for one group who's interested in you, but I wouldn't bet on that.

Referrals from your friends do little to nothing at those companies. At Microsoft, for instance, all it does is assign a "handler" to the applications you put in. They just let you know if anyone is looking at them. Which, as you can imagine, is useless.

Almost no one, except for maybe very high level hires, get a pass for the initial weed-out interviews.

Re: Ask HN: What's the state of the job market in data science and machine learning?

#107

Post by czep » Thu, Dec 22, 2016, 2:48 PM UTC

shas3 wrote: post #57

czep wrote: post #56

Earlier quoted context omitted.

> plain vanilla logistic regression Eh? It's not as easy as it sounds. Search for "mlelr" for a rather detailed illustration of how to code LR in C with only standard libraries. Now that was some fun to put together!

But that does mean that you have pretty good C programming skills! I think your comment reinforces my claim about pairing MOOC-depth ML with other skills. (Edit: I'm not saying MOOC-depth derisively. Serious learners and autodidacts can go pretty deep with just MOOCs for guidance. I just mean prima facie , based on content of some of the lighter and more popular MOOCs like Andrew Ng's. Abu-Mostafa's on EdX is meatier…

I definitely agree: the overuse of techniques like logistic regression in introductory courses without sufficient mathematical background and context can lead to a "plug and chug" culture where the technique is applied without proper understanding of its assumptions and limitations.

Anyone can be quickly taught how to run a logistic regression by calling one line of code from a high level library. But I'd argue it takes years of study to really know what you're doing.

Re: Ask HN: What's the state of the job market in data science and machine learning?

#108

Post by apohn » Thu, Dec 22, 2016, 3:14 PM UTC

Background: I currently lead a Data Science team at a big non-tech company. Previous to this I worked at a software company that had a Data Science team in their customer facing consulting group.

I'm going to speak primarily about applied data science. This means a data scientist who is solving a business need by doing ad-hoc analysis or building a reusable solution (e.g a R+Shiny dashboard) to a business problems.

Jobs: There are plenty of jobs out there, but you have to be careful. Many "Data Science" jobs are really BI, Business Analyst, or Sales Engineer types of jobs where some VP got it in their head that they need a Data Scientist. These jobs are great for people who are okay with Technology and Data Science being 10% of their job - and many people are like that. They don't care about engineering, coding, or tech and statistics beyond the minimum to do their jobs. But if you really want a job that involves solid tech and stats/ML skills you will be unsatisfied at these types of jobs.

Right now there are plenty of hard business problems that people want to turn into Data Science problems because they think it'll give them a competitive edge or something to market and show off. This results in more data science job openings. However, they are not really data science problems. As somebody else said, people will eventually realize they are not getting the value they need with data scientists doing these types of jobs. Then they'll replace that person with an MBA with some DS coursework (e.g. MBA who can use KNIME or SAS Enterprise Miner) or eliminate the position.

People: I interview people and I know people at other organizations who interview candidates for Data Science roles. MOOCs and many degree programs (including 2 year MS degrees) are pushing out people who have a very superficial overview of data science. Basically they teach them about every ML algorithm in the known universe and the functions to call them them in R/Python/SAS. The end result is a mediocre coder or non-coder who boils everything down to a confusion matrix or root mean squared error. But they cannot actually think through a business problem or see why a low error doesn't equal a good model (see http://www.tylervigen.com/spurious-correlations)

Finding good people is hard and you have to be flexible to realize great people can come from different backgrounds.

Re: Ask HN: What's the state of the job market in data science and machine learning?

#109

Post by rcpt » Thu, Dec 22, 2016, 3:41 PM UTC

wakkaflokka wrote: post #103

rm999 wrote: post #23

Speaking for NYC, but I imagine silicon valley is similar. The supply-demand dynamics have changed a lot in the last couple years. I'd roughly break it out into two groups: people with work experience + strong software development skills, and those without. The first group is in higher demand than ever, and tend to add a lot of value to companies that really need it. The second group has gotten extremely crowded, esp…

I'm coming from a PhD in STEM where I did a lot of application of basic ML to my (neuroscience) research, and it took me a good while to get a data science position. But once I got the job, I have been inundated with interview requests, both from recruiters and from specific companies, as in multiple a week (granted most are blast-em-all style recruiters, that I'm sure anybody with any tech skills get). Maybe it's be…

> a couple lines of code using SciKit Learn and all the default parameters is enough

Except for when it's not - those are the cases you're hired for.

Re: Ask HN: What's the state of the job market in data science and machine learning?

#110

Post by whenwillitstop » Thu, Dec 22, 2016, 3:42 PM UTC

shepardrtc wrote: post #106

user5994461 wrote: post #35

Earlier quoted context omitted.

You need a friend at GooMAzonSoft to refer you. It's always been and will always be the easiest way in. Then traditional uninteresting phone call and uninspired 6 hours on site with people who probably didn't read your resume. Maybe if you have a good profile and you get lucky, you'll go interview straight for one group who's interested in you, but I wouldn't bet on that.

Referrals from your friends do little to nothing at those companies. At Microsoft, for instance, all it does is assign a "handler" to the applications you put in. They just let you know if anyone is looking at them. Which, as you can imagine, is useless. Almost no one, except for maybe very high level hires, get a pass for the initial weed-out interviews.

Not true. I'm a run of the mill developer. I had another offer, I called google, told them I had another offer. They skipped all the rounds but the final.