Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We’re in the Middle of a Data Engineering Talent Shortage

blog.stitchdata.com

Post reply on HN
Print view View original

61–70 of 166 posts

Re: We’re in the Middle of a Data Engineering Talent Shortage

#61

Post by snovv_crash »

beachstartup wrote: post #37
tuna-piano wrote: post #30

Earlier quoted context omitted.

Of course this is more or less always true - there are only shortages or excesses of things when prices don't or can't adjust freely. If there was 1 gallon of water left on earth, Bill gates would buy that gallon for $50 billion, and everyone else would die of dehydration. There has always been a shortage of maids willing to do all my house work for $10. And there is a shortage of data engineers at $x, but there woul…

> If there was 1 gallon of water left on earth, Bill gates would buy that gallon for $50 billion, and everyone else would die of dehydration. really? who would sell the last gallon of water on earth?

Someone with a bunch of hydrogen, oxygen and a rudimentary knowledge of chemistry.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#62

Post by bobosha »

jnordwick wrote: post #11

Whenever I see these posts I immediate translate them in my head to "we're in the middle of a talent shortage at a price I am willing to pay." I've worked with very large amounts of data and high performance computing for most of my career; I mostly had finance related jobs in the last decade or so. I have most of the skill you want, including some you don't know you want. However when salary comes up, that is where…

>Whenever I see these posts I immediate translate them in my head to "we're in the middle of a talent shortage at a price I am willing to pay."

That's true for just about anything.

"there is no epipen crisis, only a crisis at what you are willing to pay"

"There is no poverty , only poverty at a given income level"

*"there is no crime problem, only crime problem at a given crime level"

what you are saying is self-contradictory. If you (or others) are able to turn down 150K offers...you know what you are.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#63

Post by snovv_crash »

snowwrestler wrote: post #21
jnordwick wrote: post #11

Whenever I see these posts I immediate translate them in my head to "we're in the middle of a talent shortage at a price I am willing to pay." I've worked with very large amounts of data and high performance computing for most of my career; I mostly had finance related jobs in the last decade or so. I have most of the skill you want, including some you don't know you want. However when salary comes up, that is where…

This argument comes up all the time on HN, but I don't think it means anything. It seems to me that the ability to fill an opening by offering more salary can't disprove a talent shortage, because it is always possible to do so. Thought experiment: If 100 companies had openings for a skill set that only one person could deliver, all 100 companies could eventually fill their openings by sequentially outbidding each ot…

Bullshit. I have the skills for an intermediate-level data engineer, but I find it bland and I'd rather work in computer vision. However, offer me enough and I may reconsider, and I don't think I'm alone in this.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#64

Post by [deleted] »

dizzystar wrote: post #55
mrharrison wrote: post #27

We should rename this job position to Data Sanity Engineers. I have been thrown these projects at work before, where I'm the frontend engineer and I need to make some cool D3 visualization, but low behold the data is shit, and I have to help the backend team make the data useable. It's a mind-numbing job, that nobody wants, because it sounds like a one month task to get a good REST API up and working, but it usually…

As a contradiction to this point, some people (me) really enjoy working with data, from cleaning, munging, creating, sorting, pipelining, etc, and find front-end visualization production excessively boring and mind-numbing. Give me emacs and a command line, and I have all the truth I need, which is far more honest, in my mind, than anything that can be created with D3 or Tableau. Beauty is in the eye of the beholder,…

[deleted]

Re: We’re in the Middle of a Data Engineering Talent Shortage

#65

Post by achompas »

mattnewton wrote: post #34
achompas wrote: post #20

Earlier quoted context omitted.

My official title is "Data Scientist" although I'm closer to the "ML Engineer" someone else mentions in a child comment. Frankly speaking, if your company doesn't need a data engineer, it won't hire one or move you into that role. They likely don't, either, if you're experiencing this pushback -- data engineers often develop ETL pipelines or data warehouses, both of which are very useful if your company has a data te…

I think the company does need data engineers but wants someone with a graduate degree from Stanford or CMU in that position, even though the actual work is in building up infrastructure for those people. And I understand. I've only really got software engineering skills to contribute at this point and I'm picking up the ML from kaggles on the side; I am looking for a position that can increase my overlap between thos…

I honestly fell into it by luck. I moved to NYC, studied machine learning in grad school, networked my ass off, and landed an internship.

From there I went full time as something of an ML engineer at a company with a strong tech culture, and learned as much as I could in both tech and ML/statistics. The rest is history (although I'm by no means a rockstar or whatever).

My path is hard to reproduce -- it starts with being in NYC or SF at a specific point in time, before the labor market became saturated with data science bootcamps and PhDs furiously learning Python while working on their dissertations.

Your best bet at this point is to produce a few data-related projects (maybe work on open source like scikit-learn and pandas?) and network like crazy. Someone somewhere will have a need for someone like you.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#66

Post by whenwillitstop »

jnordwick wrote: post #11

Whenever I see these posts I immediate translate them in my head to "we're in the middle of a talent shortage at a price I am willing to pay." I've worked with very large amounts of data and high performance computing for most of my career; I mostly had finance related jobs in the last decade or so. I have most of the skill you want, including some you don't know you want. However when salary comes up, that is where…

My experiences exactly, pinged by companies obsessively for my big data skills, all trying to pay me less than I am currently making.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#67

Post by mulmen »

protomyth wrote: post #53

I worked for about 10 years doing exactly what they want, but I ended up having to write a lot of the tools which means I'm not able to check the boxes on some tool you require which gets me punted by HR. I'm starting to think that the message is if HR is going to do checklists then developers should really make sure they work mostly with contracts that use popular checklist items.

As a data person I would really like to put some numbers on how much the typical HR hiring process costs a business. I don't know anybody that says they are happy with how hiring works in he tech industry but I've also never seen an HR person try and improve the process.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#68

Post by stale2002 »

jnordwick wrote: post #11

Whenever I see these posts I immediate translate them in my head to "we're in the middle of a talent shortage at a price I am willing to pay." I've worked with very large amounts of data and high performance computing for most of my career; I mostly had finance related jobs in the last decade or so. I have most of the skill you want, including some you don't know you want. However when salary comes up, that is where…

Although your statement is technically true, it is basically meaningless.

Yes, you can always always always find somebody to do a job is your a willing to pay 10 million dollars. That means that "shortages" are impossible. It means that you can never have a shortage in any situation, because you can always pay 10 million dollars for a single visit to the doctor.

But this line of logic isn't very useful when talking about "shortages".

If you had to pay a million dollars for a loaf of bread, is there a shortage of bread? IE, billions of people will starve to death by next week, because they can't afford to buy food.

Most people would say "Yes, there is a shortage of bread".

When people talk about shortages, they are obviously talking about a shortage at a certain price point. There is no other definition of the word shortage that makes sense.

A good definition that I use for the term shortage is "If the government could snap its fingers and instantly produce large amounts of X overnight, would the world be a better place"?

If the answer is "Yes, the world would be in a much much better place", then that means there is a shortage of X. If the answer is "No, the world would only be a little better". Then that means that there is NOT a shortage of X.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#69

Post by SmellTheGlove »

jnordwick wrote: post #11

Whenever I see these posts I immediate translate them in my head to "we're in the middle of a talent shortage at a price I am willing to pay." I've worked with very large amounts of data and high performance computing for most of my career; I mostly had finance related jobs in the last decade or so. I have most of the skill you want, including some you don't know you want. However when salary comes up, that is where…

I think it's definitely true. Functionally, I'm a director of data engineering (with a big company, so my real title is way more generic). Usually in the initial screen, we'll talk general dollars, and my number is always out of range. For my level and the fact that I'm reasonably happy where I live now, the number is 200k + relocation (more for Bay Area, but lets not go there), and I don't think that's unreasonable for a director level who is presumably going to also develop your more junior DEs.

I don't fancy it up too much, either. I build teams that make the data move and land it clean so that your PhDs can do the smaaaht stuff with it. I can stack BI and Analytics on top, but a lot of people can do that starting from clean data - and clean data is what I do. But I do get the impression that we're viewed as janitors and plumbers - who you'd be thrilled to see at 3am when your shit(ter) broke, right?

Re: We’re in the Middle of a Data Engineering Talent Shortage

#70

Post by blahi »

trendia wrote: post #25
snowwrestler wrote: post #21

Earlier quoted context omitted.

This argument comes up all the time on HN, but I don't think it means anything. It seems to me that the ability to fill an opening by offering more salary can't disprove a talent shortage, because it is always possible to do so. Thought experiment: If 100 companies had openings for a skill set that only one person could deliver, all 100 companies could eventually fill their openings by sequentially outbidding each ot…

In this case, though, there are way more than 6,600 people in the US that would be able to get do that data engineering job, including: 1. physicists 2. Wall St. quants 3. game programmers 4. PhD statisticians So, the problem is not that there aren't 6,600 people in the US that can do it, it's that the companies can't pay or don't want to pay the $200,000 + that would be required to hire them.

phd statistician can write ETLs and data infrastructure?
Post reply on HN