Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We’re in the Middle of a Data Engineering Talent Shortage

blog.stitchdata.com

Post reply on HN
Print view View original

21–30 of 166 posts

Re: We’re in the Middle of a Data Engineering Talent Shortage

#21

Post by snowwrestler »

jnordwick wrote: post #11

Whenever I see these posts I immediate translate them in my head to "we're in the middle of a talent shortage at a price I am willing to pay." I've worked with very large amounts of data and high performance computing for most of my career; I mostly had finance related jobs in the last decade or so. I have most of the skill you want, including some you don't know you want. However when salary comes up, that is where…

This argument comes up all the time on HN, but I don't think it means anything. It seems to me that the ability to fill an opening by offering more salary can't disprove a talent shortage, because it is always possible to do so.

Thought experiment: If 100 companies had openings for a skill set that only one person could deliver, all 100 companies could eventually fill their openings by sequentially outbidding each other for the services of that one person.

So how would we know if a talent shortage really exists for a certain job? I can think of a couple potential hints: if starting salaries are going up much faster than the national average, or if the unemployment rate for that job is much lower than the national unemployment rate. Either would seem to indicate that, relative to the job market as a whole, there was a greater demand than supply for that particular job.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#22

Post by achompas »

jnordwick wrote: post #11

Whenever I see these posts I immediate translate them in my head to "we're in the middle of a talent shortage at a price I am willing to pay." I've worked with very large amounts of data and high performance computing for most of my career; I mostly had finance related jobs in the last decade or so. I have most of the skill you want, including some you don't know you want. However when salary comes up, that is where…

Yeah, this is a selling problem. It feels like you're far more likely to gain traction starting a data team than taking an IC-track DE role. It's easier for companies to justify $200k+ for your skillset in that case, even if it takes you away from pure engineering.

Alternatively, you can just join a large tech org. Netflix etc. have no problem paying good DEs north of $200k in total comp.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#24

Post by mattnewton »

PaulHoule wrote: post #5
mattnewton wrote: post #2

I'm trying to switch careers into "Data Engineering" now, as a full stack developer who is more interested in ML, and I've found almost no traction internally at my company or externally. It looks like I may just accept a full stack position at a good company that does a lot of data science for now, but though I would ask - Where are all these jobs?

"Data Engineering" is most of the work that needs to be done, but I think companies haven't identified it as a category. From my P.O.V., "Full Stack Engineer" is a place you don't want to be because it means putting out fires with whatever junk javascript is in the front end. It seems like everybody who's built a serious javascript application has invented their own Virtual DOM because none of the popular Virtual DOM…

You are correct. I thought full stack meant before building the app start to finish, but the reality is often closer to putting out other people's fires in every layer. It does pay well though and you learn a lot of what can go wrong.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#25

Post by trendia »

snowwrestler wrote: post #21
jnordwick wrote: post #11

Whenever I see these posts I immediate translate them in my head to "we're in the middle of a talent shortage at a price I am willing to pay." I've worked with very large amounts of data and high performance computing for most of my career; I mostly had finance related jobs in the last decade or so. I have most of the skill you want, including some you don't know you want. However when salary comes up, that is where…

This argument comes up all the time on HN, but I don't think it means anything. It seems to me that the ability to fill an opening by offering more salary can't disprove a talent shortage, because it is always possible to do so. Thought experiment: If 100 companies had openings for a skill set that only one person could deliver, all 100 companies could eventually fill their openings by sequentially outbidding each ot…

In this case, though, there are way more than 6,600 people in the US that would be able to get do that data engineering job, including:

1. physicists

2. Wall St. quants

3. game programmers

4. PhD statisticians

So, the problem is not that there aren't 6,600 people in the US that can do it, it's that the companies can't pay or don't want to pay the $200,000 + that would be required to hire them.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#26

Post by sdoowpilihp »

jnordwick wrote: post #11

Whenever I see these posts I immediate translate them in my head to "we're in the middle of a talent shortage at a price I am willing to pay." I've worked with very large amounts of data and high performance computing for most of my career; I mostly had finance related jobs in the last decade or so. I have most of the skill you want, including some you don't know you want. However when salary comes up, that is where…

This has been my experience with any "senior" engineering / BI / DS role. There is a particularly high level of price sensitivity to anything above 200k.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#27

Post by mrharrison »

We should rename this job position to Data Sanity Engineers.

I have been thrown these projects at work before, where I'm the frontend engineer and I need to make some cool D3 visualization, but low behold the data is shit, and I have to help the backend team make the data useable. It's a mind-numbing job, that nobody wants, because it sounds like a one month task to get a good REST API up and working, but it usually takes three months, because you have to go back and forth making sure the data is right, and there is always 10 tricky edge cases that you have to work some magic on. Not only that but you need to have smart people cleaning the data, so that you don't make some big mistake down the line or your REST API is super slow, and you have to add another couple weeks or month to rework the data again. So that one month becomes three months, and most likely a year, because somebody will say that looks great but can we also add this, and it goes on and on. It's literally a mind-numbing job that most nobody wants. I have found that products like Tableau are the best for this, you still have to clean the data, but it helps speed up the process.

Data cleaning is a super golden problem to solve.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#28

Post by mattnewton »

willis77 wrote: post #6
mattnewton wrote: post #2

I'm trying to switch careers into "Data Engineering" now, as a full stack developer who is more interested in ML, and I've found almost no traction internally at my company or externally. It looks like I may just accept a full stack position at a good company that does a lot of data science for now, but though I would ask - Where are all these jobs?

We (Kaggle) run a data science jobs board ( https://www.kaggle.com/jobs ) that gets a few data engineer listings from time to time. Not all of these are active, but you may find a few interested companies via - https://www.google.com/#q=site:https://www.kaggle.com/jobs+%...

Thank you guys! Doing Kaggle competitions is what got me interested in seriously pursuing ML in the first place. You are all seriously awesome.

I'll look again at the board but, I didn't see anything there before that wanted software engineering skills (which I have with industry experience), and not a graduate degree (which I don't), and happened to be commutable from my place just south of the bay. But I will keep looking!

Re: We’re in the Middle of a Data Engineering Talent Shortage

#29

Post by pavlov »

dmatthewson wrote: post #16

From the article: "Data engineers are the janitors who keep your data clean and flowing." Hm, I wonder why he's having problems hiring janitors.

Bizarrely, I remember a recent HN discussion where a poster was arguing that any software developer who is not working in machine learning is like a plumber.

I guess this means that the entire profession consists of janitors and plumbers.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#30

Post by tuna-piano »

jnordwick wrote: post #11

Whenever I see these posts I immediate translate them in my head to "we're in the middle of a talent shortage at a price I am willing to pay." I've worked with very large amounts of data and high performance computing for most of my career; I mostly had finance related jobs in the last decade or so. I have most of the skill you want, including some you don't know you want. However when salary comes up, that is where…

Of course this is more or less always true - there are only shortages or excesses of things when prices don't or can't adjust freely.

If there was 1 gallon of water left on earth, Bill gates would buy that gallon for $50 billion, and everyone else would die of dehydration.

There has always been a shortage of maids willing to do all my house work for $10.

And there is a shortage of data engineers at $x, but there wouldn't be a shortage at $1M/year (because less companies would want one, and more people would be willing to do the work).

Post reply on HN