We’re in the Middle of a Data Engineering Talent Shortage • Hacker News Board

Re: We’re in the Middle of a Data Engineering Talent Shortage

#141

Post by beachstartup » Fri, Sep 09, 2016, 8:58 AM UTC

moandcompany wrote: post #41

I am a data engineer working on a machine learning team with models actively used as part of our product(s). From my experiences working in various contexts (applied machine learning, analytics, policy research, academics, etc...), there are several of factors that contribute to this shortage: (1) "data engineering" often requires a lot of breadth and knowledge, (2) "data engineering" is often (derisively and naively…

in my mind the problem is really simple: most executives aren't smart enough to understand how any of this shit works, or build a compelling business case around it. they just know they need a 'big data' team, so it just dies on the vine.

someone with enough smarts to build/lead a team, sell to executive management, and have an actual business application is just too rare compared to the prevalence of the engineering talent.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#142

Post by ThomPete » Fri, Sep 09, 2016, 1:10 PM UTC

duaneb wrote: post #47

ThomPete wrote: post #18

Earlier quoted context omitted.

Upskilling is one of the most ineffective costly ways to try and "re-program" workers and it mostly doesn't work because it's not about skills it's about talent.

How would you distinguish talent from experience?

I would say that talent needs experience to be valuable.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#143

Post by hga » Fri, Sep 09, 2016, 1:37 PM UTC

jballanc wrote: post #133

pavlov wrote: post #29

Earlier quoted context omitted.

Bizarrely, I remember a recent HN discussion where a poster was arguing that any software developer who is not working in machine learning is like a plumber. I guess this means that the entire profession consists of janitors and plumbers.

Considering that plumbers and janitors have likely, in the entire history of human civilization, done more for health and longevity than doctors and scientists...I'm kind of ok with this analogy.

Doctors, maybe, but it was the scientists who told them about the germ theory of disease, for instance.

I've read, but not confirmed for myself, that in the US the biggest gains in health came in the post-Civil War period, when "plumbers and janitors" made the difference. Of course, that's really starting with, after the science, the civil engineers who designed the public works systems that supplied clean water and took away sewage, and let's not forget that politicians and like who found it worthwhile to buy votes that way (now, they take our infrastructure for granted and buy votes more directly...).

Re: We’re in the Middle of a Data Engineering Talent Shortage

#144

Post by ThomPete » Fri, Sep 09, 2016, 1:43 PM UTC

crispyambulance wrote: post #81

ThomPete wrote: post #18

Earlier quoted context omitted.

Upskilling is one of the most ineffective costly ways to try and "re-program" workers and it mostly doesn't work because it's not about skills it's about talent.

I sort of agree in that orgs can't simply create massive education programs to re-purpose skill-sets/talent. That might have been possible "back in the day" before project managers were breathing down people's necks, but not today. But the brightside is that talented people will find a way to "upskill" themselves in whatever environment they find themselves in. It is then up to the candidates to sell themselves and f…

Yeah if up-skilling means "I used to do front end development now I do back end development" the up-skilling is fairly easy to do.

But if you used to work as a plumber and want to up-skill to data analyst (or vice versa) it's not that simple.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#145

Post by ThomPete » Fri, Sep 09, 2016, 1:51 PM UTC

etjossem wrote: post #104

ThomPete wrote: post #18

Earlier quoted context omitted.

Upskilling is one of the most ineffective costly ways to try and "re-program" workers and it mostly doesn't work because it's not about skills it's about talent.

That's not always the case. Talent doesn't exist in a vacuum. Someone with a natural talent for picking up new development skills will still learn data engineering far faster when provided with proper resources and strong internal mentorship. I can see how you might make this observation after observing a poorly conducted training program.

The problem is that there aren't that many people with natural talents. They exist but it's very hard to sync them up with where demand is.

Also this is not just one poorly conducted training program. Denmark spent billions up-skilling parts of their work force. The results where simply no there. Something like 6 out of every 1000 person or something like that.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#146

Post by infinite8s » Fri, Sep 09, 2016, 2:24 PM UTC

Declanomous wrote: post #71

trendia wrote: post #25

Earlier quoted context omitted.

In this case, though, there are way more than 6,600 people in the US that would be able to get do that data engineering job, including: 1. physicists 2. Wall St. quants 3. game programmers 4. PhD statisticians So, the problem is not that there aren't 6,600 people in the US that can do it, it's that the companies can't pay or don't want to pay the $200,000 + that would be required to hire them.

This comment will sound a bit self-serving, but it supports your point. I have most of the skills necessary to be a data engineer. My degree in biology, but I nearly got a double major in computer science with a minor in math. (I wanted to work in bioinformatics, but it's nearly impossible to make more than a pittance without a PhD) I didn't pursue the double major because I felt taking classes outside of those three…

> (I wanted to work in bioinformatics, but it's nearly impossible to make more than a pittance without a PhD)

Just wanted to comment on this part - sadly it's difficult to make more than a pittance even with a PhD.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#147

Post by wpiel » Fri, Sep 09, 2016, 2:31 PM UTC

What I've learned from the comments: If something is valuable, there is a shortage of it.

I'm not even sure if I'm being sarcastic.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#148

Post by LawrenceHecht » Fri, Sep 09, 2016, 2:37 PM UTC

Just checked, the # of data engineers rose to 9,246 (42%) in the last six months. So, the shortage is at least being addressed by people changing their job titles on LinkedIn.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#149

Post by infinite8s » Fri, Sep 09, 2016, 3:03 PM UTC

p4wnc6 wrote: post #43

achompas wrote: post #20

Earlier quoted context omitted.

My official title is "Data Scientist" although I'm closer to the "ML Engineer" someone else mentions in a child comment. Frankly speaking, if your company doesn't need a data engineer, it won't hire one or move you into that role. They likely don't, either, if you're experiencing this pushback -- data engineers often develop ETL pipelines or data warehouses, both of which are very useful if your company has a data te…

> There's actually a shortage of data-savvy people who can also write production software, and you would nicely complement a more research-inclined data scientist or analyst -- someone with far more experience with research/analysis than development. I experience the same problem with shortage-at-price-X in the field you describe. I'm a machine learning engineer with experience in MCMC methods, but I also have a lot…

I suspect the reason is those companies offering $140k frankly don't need that level of expertise. With that kind of background it would be fairly easy to get 200-300k as an infrastructure engineer at a quant shop.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#150

Post by geebee » Fri, Sep 09, 2016, 3:29 PM UTC

p4wnc6 wrote: post #43

achompas wrote: post #20

Earlier quoted context omitted.

My official title is "Data Scientist" although I'm closer to the "ML Engineer" someone else mentions in a child comment. Frankly speaking, if your company doesn't need a data engineer, it won't hire one or move you into that role. They likely don't, either, if you're experiencing this pushback -- data engineers often develop ETL pipelines or data warehouses, both of which are very useful if your company has a data te…

> There's actually a shortage of data-savvy people who can also write production software, and you would nicely complement a more research-inclined data scientist or analyst -- someone with far more experience with research/analysis than development. I experience the same problem with shortage-at-price-X in the field you describe. I'm a machine learning engineer with experience in MCMC methods, but I also have a lot…

"I've seen reputable, multi-billion dollar companies offering in the $140k range for this type of role in New York. That's wildly below anything reasonable for this sort of thing [in NY/SF"]

Man, do I ever agree. This is where the "shortage" argument falls apart.

This is why I'm so uninterested in the abstract arguments happening elsewhere on this topic about whether markets are failing and basic laws of supply and demand no longer apply at theoretical salary levels (10 million was offered as an example).

Why are we bothering with this debate, when it's so far from reality? I'd say that if you're trying to hire a very high skilled and critical tech worker in SF, and you just can't find one no matter how hard you try, and then I find out that you're only offering 140k a year?

In San Francisco and New York (and anywhere else in the US, really), that's nowhere close to the kind of pay where we should start scratching our heads about a shortage and start wondering why the usual laws of supply and demand aren't working anymore.