We’re in the Middle of a Data Engineering Talent Shortage • Hacker News Board

Re: We’re in the Middle of a Data Engineering Talent Shortage

#101

Post by vijayr » Thu, Sep 08, 2016, 8:32 PM UTC

moandcompany wrote: post #41

I am a data engineer working on a machine learning team with models actively used as part of our product(s). From my experiences working in various contexts (applied machine learning, analytics, policy research, academics, etc...), there are several of factors that contribute to this shortage: (1) "data engineering" often requires a lot of breadth and knowledge, (2) "data engineering" is often (derisively and naively…

How does one get started with this? I suppose a lot of people who hang out at HN are competent devs good in programming and databases, but probably beginners in math, ML, AI etc. How does such a person get started and find a job in this field?

Re: We’re in the Middle of a Data Engineering Talent Shortage

#102

Post by collyw » Thu, Sep 08, 2016, 8:33 PM UTC

otto_ortega wrote: post #23

Am I the only one who thinks there will be a ton of people changing their job title on LinkedIn to "Data Engineer" as a result of this article?

I am thinking about it. Actually a friend recommended that I change my title to Data Engineer a few months back.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#103

Post by jnordwick » Thu, Sep 08, 2016, 8:35 PM UTC

stale2002 wrote: post #68

jnordwick wrote: post #11

Whenever I see these posts I immediate translate them in my head to "we're in the middle of a talent shortage at a price I am willing to pay." I've worked with very large amounts of data and high performance computing for most of my career; I mostly had finance related jobs in the last decade or so. I have most of the skill you want, including some you don't know you want. However when salary comes up, that is where…

Although your statement is technically true, it is basically meaningless. Yes, you can always always always find somebody to do a job is your a willing to pay 10 million dollars. That means that "shortages" are impossible. It means that you can never have a shortage in any situation, because you can always pay 10 million dollars for a single visit to the doctor. But this line of logic isn't very useful when talking a…

If they were offering $300k and still couldn't find attract top talent from other industries, then we could have a discussion about a shortage, but the low six figures doesn't show a shortage situation. It shows a market with plenty of headroom for salaries to grow.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#104

Post by etjossem » Thu, Sep 08, 2016, 8:36 PM UTC

ThomPete wrote: post #18

tobyjsullivan wrote: post #14

Earlier quoted context omitted.

"we're in the middle of a talent shortage [and don't believe in upskilling]."

Upskilling is one of the most ineffective costly ways to try and "re-program" workers and it mostly doesn't work because it's not about skills it's about talent.

That's not always the case. Talent doesn't exist in a vacuum.

Someone with a natural talent for picking up new development skills will still learn data engineering far faster when provided with proper resources and strong internal mentorship.

I can see how you might make this observation after observing a poorly conducted training program.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#105

Post by whybroke » Thu, Sep 08, 2016, 8:39 PM UTC

stale2002 wrote: post #68

jnordwick wrote: post #11

Whenever I see these posts I immediate translate them in my head to "we're in the middle of a talent shortage at a price I am willing to pay." I've worked with very large amounts of data and high performance computing for most of my career; I mostly had finance related jobs in the last decade or so. I have most of the skill you want, including some you don't know you want. However when salary comes up, that is where…

Although your statement is technically true, it is basically meaningless. Yes, you can always always always find somebody to do a job is your a willing to pay 10 million dollars. That means that "shortages" are impossible. It means that you can never have a shortage in any situation, because you can always pay 10 million dollars for a single visit to the doctor. But this line of logic isn't very useful when talking a…

There is no analogy to bread or 10miilon dollar salaries.

A company found something it could profit from more if it paid less than current market value. That is all. They are not saying there are no qualified applicants. They are not saying they want 10million dollars.

What is the case is that a business finds a resource (the perfect hire) that they wish to profit from but do not want to pay the market value for it because that would reduce profits. Rather than be satisfied with what would be an erosion of profit (or an admission of an unworkable business model) articles are posted to demand government pressure wages downward.

If you want a bread analogy, it's as if I found a cheap source of bread I can sell elsewhere at a profit but then complain there's a shortage solely because the cheap stuff isn't even cheaper.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#106

Post by biztos » Thu, Sep 08, 2016, 8:44 PM UTC

binalpatel wrote: post #99

The fact that the original, unmodified article referred to data engineers as "janitors" pretty much says it all. It's very analogous to front-office and back-office work in Investment Banking. "Data Scientist" are the front-office, with all the prestige, and "Data Engineers" are the back-office, doing a lot of the heavy lifting without nearly as much recognition. In my opinion there shouldn't be a delineation. You sh…

Ideally you'd have a symbiosis, and each side would recognize the importance of the other.

Even if you require your data scientists to be able to do engineering work, it's probably way more efficient to have some good generalist Software Engineers doing all the "pre-math" work and freeing your statisticians up for what they're (hopefully) good at.

Plus as a side effect, your software will probably be better.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#107

Post by sportanova » Thu, Sep 08, 2016, 8:47 PM UTC

GeneralMayhem wrote: post #15

There are 6600 jobs listed and 6500 individuals on LinkedIn with that particular title, and therefore there's a shortage? Seriously? * How many aren't on LinkedIn? * Since the whole article is about how the job title is poorly defined and growing in prevalence, why would you assume that people who don't already have such a job would use the term? * The "growth" charts on the full study are just as bad - how much of t…

6500 data engineers on all of Linkedin, but 6600 job openings in the bay area. so there are more job openings in one area than all data engineers on linkedin

Re: We’re in the Middle of a Data Engineering Talent Shortage

#108

Post by huac » Thu, Sep 08, 2016, 8:47 PM UTC

walshemj wrote: post #95

huac wrote: post #73

Earlier quoted context omitted.

I'm going to guess most of those people couldn't set up and scale a Hadoop cluster. Are they smart enough that they could learn this stuff? Sure! But there's still a skill mismatch here.

so you find some real full stack devs ie layers 1-7 or you buckle down and do a CCNA or similar is there a CCIE track for big data? And I would bet across all of the tech workers in the USA there are well more than 6k that could do this.

So, obviously a LinkedIn search for exact title of "data engineer" isn't exhaustive. And as I understand your point, there's certainly no agreed upon group of skills/certification that qualify someone to be a data engineer (or data scientist, or software engineer, for that matter).

But the GP was particularly amusing to me because of its assertion that 'smart, quantitative people, regardless of industry, can build data infrastructures for startups.' I guess we could also say, there's little incentive to pay to train them (or for them to pay to train) to become a data engineer.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#109

Post by PaulHoule » Thu, Sep 08, 2016, 8:48 PM UTC

mattnewton wrote: post #24

PaulHoule wrote: post #5

Earlier quoted context omitted.

"Data Engineering" is most of the work that needs to be done, but I think companies haven't identified it as a category. From my P.O.V., "Full Stack Engineer" is a place you don't want to be because it means putting out fires with whatever junk javascript is in the front end. It seems like everybody who's built a serious javascript application has invented their own Virtual DOM because none of the popular Virtual DOM…

You are correct. I thought full stack meant before building the app start to finish, but the reality is often closer to putting out other people's fires in every layer. It does pay well though and you learn a lot of what can go wrong.

The fact that it pays well makes it a job you're likely to get laid off from. Most managers would rather hire two junior developers so they can screw it up faster or better yet hire some people in another country who are really fast and cheap at screwing it up.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#110

Post by biztos » Thu, Sep 08, 2016, 8:50 PM UTC

pavlov wrote: post #29

dmatthewson wrote: post #16

From the article: "Data engineers are the janitors who keep your data clean and flowing." Hm, I wonder why he's having problems hiring janitors.

Bizarrely, I remember a recent HN discussion where a poster was arguing that any software developer who is not working in machine learning is like a plumber. I guess this means that the entire profession consists of janitors and plumbers.

I recently had a plumber do some work on a >100-year-old apartment. I was lucky: he's a very good plumber.

The job didn't involve too many "pipelines" but the knowledge and creativity required to make them work was well above what I see from most software developers.

"Plumber" is not the put-down that poster thought it was.