Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We’re in the Middle of a Data Engineering Talent Shortage

blog.stitchdata.com

Post reply on HN
Print view View original

91–100 of 166 posts

Re: We’re in the Middle of a Data Engineering Talent Shortage

#91

Post by blihp »

dizzystar wrote: post #78
jnordwick wrote: post #11

Whenever I see these posts I immediate translate them in my head to "we're in the middle of a talent shortage at a price I am willing to pay." I've worked with very large amounts of data and high performance computing for most of my career; I mostly had finance related jobs in the last decade or so. I have most of the skill you want, including some you don't know you want. However when salary comes up, that is where…

Are they looking for someone who must have every box ticked or are they looking for someone with enough qualifications yet needing work so much they are willing to undercut themselves? Are they justifying their salary offer because you tick 90% of the boxes and not 100%? I've been looking for work in data engineering and databases for 9 months, and while I'm certainly not as qualified and experienced as you are, I co…

Of course they must. By demonstrating that they cannot find candidates with 100% of the 'required' skills at the price they are willing to pay, the path is cleared to go the route of 'highly skilled' H1B applicants etc. with a small percentage of these skills. It is not, and has never been, about the skills.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#92

Post by walshemj »

SmellTheGlove wrote: post #89
dizzystar wrote: post #78

Earlier quoted context omitted.

Are they looking for someone who must have every box ticked or are they looking for someone with enough qualifications yet needing work so much they are willing to undercut themselves? Are they justifying their salary offer because you tick 90% of the boxes and not 100%? I've been looking for work in data engineering and databases for 9 months, and while I'm certainly not as qualified and experienced as you are, I co…

That to me is a classic recruiting problem in technical positions, data engineering included. Unless you have a manager handling it themselves, the person doing the initial screen really is ticking boxes because they may not know any better. Once a resume gets to me, and I'm only speaking for myself here, I'm looking for the challenges you've faced and the problems you've solved. I actually care very little about wha…

Quite I am sure my experience doing MR back in the early 80's for British Telecom would be usefull today - but I suspect that I might struggle to get past the hr screen.

That was when 17 top of the line supermini's Pr!me 750's was a huge cluster (we where the largest non back user in the UK) - probably about the same as a 10-20k core Hadoop setup would be today.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#93

Post by collyw »

So I know SQL, Python, Django, Java (though its been a while), Javascrit, Linux, some cloud computing and a bit of devops. Am I a data engineer? Software engineer, with a lot of database background? What makes a data engineer different from a software engineer?

Re: We’re in the Middle of a Data Engineering Talent Shortage

#94

Post by sloxy »

SmellTheGlove wrote: post #69
jnordwick wrote: post #11

Whenever I see these posts I immediate translate them in my head to "we're in the middle of a talent shortage at a price I am willing to pay." I've worked with very large amounts of data and high performance computing for most of my career; I mostly had finance related jobs in the last decade or so. I have most of the skill you want, including some you don't know you want. However when salary comes up, that is where…

I think it's definitely true. Functionally, I'm a director of data engineering (with a big company, so my real title is way more generic). Usually in the initial screen, we'll talk general dollars, and my number is always out of range. For my level and the fact that I'm reasonably happy where I live now, the number is 200k + relocation (more for Bay Area, but lets not go there), and I don't think that's unreasonable…

"director of data engineering"

This is already generic.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#95

Post by walshemj »

huac wrote: post #73
trendia wrote: post #25

Earlier quoted context omitted.

In this case, though, there are way more than 6,600 people in the US that would be able to get do that data engineering job, including: 1. physicists 2. Wall St. quants 3. game programmers 4. PhD statisticians So, the problem is not that there aren't 6,600 people in the US that can do it, it's that the companies can't pay or don't want to pay the $200,000 + that would be required to hire them.

I'm going to guess most of those people couldn't set up and scale a Hadoop cluster. Are they smart enough that they could learn this stuff? Sure! But there's still a skill mismatch here.

so you find some real full stack devs ie layers 1-7 or you buckle down and do a CCNA or similar is there a CCIE track for big data?

And I would bet across all of the tech workers in the USA there are well more than 6k that could do this.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#96

Post by mrharrison »

dizzystar wrote: post #55
mrharrison wrote: post #27

We should rename this job position to Data Sanity Engineers. I have been thrown these projects at work before, where I'm the frontend engineer and I need to make some cool D3 visualization, but low behold the data is shit, and I have to help the backend team make the data useable. It's a mind-numbing job, that nobody wants, because it sounds like a one month task to get a good REST API up and working, but it usually…

As a contradiction to this point, some people (me) really enjoy working with data, from cleaning, munging, creating, sorting, pipelining, etc, and find front-end visualization production excessively boring and mind-numbing. Give me emacs and a command line, and I have all the truth I need, which is far more honest, in my mind, than anything that can be created with D3 or Tableau. Beauty is in the eye of the beholder,…

So I guess you are a data engineer? What makes it fun for you? How do work with your customers to give them what they need in a timely matter? I would be interested to know what stack you use to go from dirty data to customer consumption.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#97

Post by ronald_raygun »

blahi wrote: post #70
trendia wrote: post #25

Earlier quoted context omitted.

In this case, though, there are way more than 6,600 people in the US that would be able to get do that data engineering job, including: 1. physicists 2. Wall St. quants 3. game programmers 4. PhD statisticians So, the problem is not that there aren't 6,600 people in the US that can do it, it's that the companies can't pay or don't want to pay the $200,000 + that would be required to hire them.

phd statistician can write ETLs and data infrastructure?

There can be a lot of trickiness in this. I worked on A/B testing framework at one of the big software cos and me and all my team had a masters or phd in math or stats. While 95+% of our job was data infrastructure and ETLs there is another dimension to making it work and be correct from a statistical point of view.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#98

Post by jboggan »

It's digital Charlie Work [0], that's why.

I really enjoy that kind of work but it is difficult to articulate your business value in that environment. The best thing is working closely with a data scientist/front-end dev who can deliver products to the analysts and executives that need the data and make sure that you get the credit for enabling new streams of data. But most of the time you are putting out someone else's dumpster fire.

One advantage of data engineering: unlike front-end work, there are few non-technical people who will have an opinion on how you are doing things and burden you with bikeshedding.

[0] - http://www.avclub.com/tvclub/its-always-sunny-philadelphia-c...

Re: We’re in the Middle of a Data Engineering Talent Shortage

#99

Post by binalpatel »

The fact that the original, unmodified article referred to data engineers as "janitors" pretty much says it all.

It's very analogous to front-office and back-office work in Investment Banking. "Data Scientist" are the front-office, with all the prestige, and "Data Engineers" are the back-office, doing a lot of the heavy lifting without nearly as much recognition.

In my opinion there shouldn't be a delineation. You shouldn't be a data scientist if you can't gather, process, and clean up your own data.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#100

Post by kafkaesq »

wavefunction wrote: post #86
kafkaesq wrote: post #76

Earlier quoted context omitted.

So is the work that doctors, lawyers, and other highly-skilled people do, by and large. Everyone knows that day-to-day aspects of these jobs are hardly glamorous (or even cerebral), the vast majority of the time. Yet somehow we manage to accord these people with their due degree of respect, and wouldn't think of referring to them as "janitors".

I don't see what's so bad about janitors, though. They do very thankless jobs for not much money, whereas doctors and lawyers and other high-skilled individuals are often well renumerated or offered certain social prestige that your post shows is quite lacking when a humble janitor is considered.

A perfectly valid point. But getting back to the original article -- it pretty much takes a SV alpha-nerd (or aspiring CEO seeking to cater to them) to come up with language like that.
Post reply on HN