We’re in the Middle of a Data Engineering Talent Shortage • Hacker News Board

Re: We’re in the Middle of a Data Engineering Talent Shortage

#111

Post by walshemj » Thu, Sep 08, 2016, 8:51 PM UTC

huac wrote: post #108

walshemj wrote: post #95

Earlier quoted context omitted.

so you find some real full stack devs ie layers 1-7 or you buckle down and do a CCNA or similar is there a CCIE track for big data? And I would bet across all of the tech workers in the USA there are well more than 6k that could do this.

So, obviously a LinkedIn search for exact title of "data engineer" isn't exhaustive. And as I understand your point, there's certainly no agreed upon group of skills/certification that qualify someone to be a data engineer (or data scientist, or software engineer, for that matter). But the GP was particularly amusing to me because of its assertion that 'smart, quantitative people, regardless of industry, can build da…

Ah the British Disease ie don't want to pay for training and we don't want those uppity engineers getting above themselves :-(

Re: We’re in the Middle of a Data Engineering Talent Shortage

#112

Post by mattnewton » Thu, Sep 08, 2016, 8:56 PM UTC

PaulHoule wrote: post #109

mattnewton wrote: post #24

Earlier quoted context omitted.

You are correct. I thought full stack meant before building the app start to finish, but the reality is often closer to putting out other people's fires in every layer. It does pay well though and you learn a lot of what can go wrong.

The fact that it pays well makes it a job you're likely to get laid off from. Most managers would rather hire two junior developers so they can screw it up faster or better yet hire some people in another country who are really fast and cheap at screwing it up.

That may be true but I'm not worried about that, I worry more about getting comfortable doing useless work. If I got fired it would be so much easier to go back to school, as the dream of lots of money while learning on the side would evaporate.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#113

Post by biztos » Thu, Sep 08, 2016, 8:58 PM UTC

jnordwick wrote: post #11

Whenever I see these posts I immediate translate them in my head to "we're in the middle of a talent shortage at a price I am willing to pay." I've worked with very large amounts of data and high performance computing for most of my career; I mostly had finance related jobs in the last decade or so. I have most of the skill you want, including some you don't know you want. However when salary comes up, that is where…

Maybe you should start a Data Science & Engineering consultancy. The same people who would offer $150K to an employee often have bosses who would love to spend $500K for a person-year of (contract) work if it comes with a high probability of success.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#114

Post by mattnewton » Thu, Sep 08, 2016, 8:58 PM UTC

achompas wrote: post #65

mattnewton wrote: post #34

Earlier quoted context omitted.

I think the company does need data engineers but wants someone with a graduate degree from Stanford or CMU in that position, even though the actual work is in building up infrastructure for those people. And I understand. I've only really got software engineering skills to contribute at this point and I'm picking up the ML from kaggles on the side; I am looking for a position that can increase my overlap between thos…

I honestly fell into it by luck. I moved to NYC, studied machine learning in grad school, networked my ass off, and landed an internship. From there I went full time as something of an ML engineer at a company with a strong tech culture, and learned as much as I could in both tech and ML/statistics. The rest is history (although I'm by no means a rockstar or whatever). My path is hard to reproduce -- it starts with b…

Thanks! I guess it's somewhat reassuring that it's hard to break into for everyone and I'm not just dumb :) I'll keep kaggin'

Re: We’re in the Middle of a Data Engineering Talent Shortage

#115

Post by branchless » Thu, Sep 08, 2016, 9:02 PM UTC

beachstartup wrote: post #37

tuna-piano wrote: post #30

Earlier quoted context omitted.

Of course this is more or less always true - there are only shortages or excesses of things when prices don't or can't adjust freely. If there was 1 gallon of water left on earth, Bill gates would buy that gallon for $50 billion, and everyone else would die of dehydration. There has always been a shortage of maids willing to do all my house work for $10. And there is a shortage of data engineers at $x, but there woul…

> If there was 1 gallon of water left on earth, Bill gates would buy that gallon for $50 billion, and everyone else would die of dehydration. really? who would sell the last gallon of water on earth?

someone with liquid assets. I'll get my coat...

Re: We’re in the Middle of a Data Engineering Talent Shortage

#116

Post by jakestein » Thu, Sep 08, 2016, 9:08 PM UTC

wavefunction wrote: post #87

jakestein wrote: post #59

Earlier quoted context omitted.

Hey I'm the author of this blog post and the CEO of the company that did the benchmark report. That was a very poor choice of words on my part, and I appreciate you flagging it. I reworked the paragraph to remove the janitor comment and (hopefully) make it clearer.

You should also not use "janitor" as a disparaging term. That would be another good takeaway from all of this.

I agree that it's a bad idea to use "janitor" as a disparaging term, and that was very far away from my intention. If that was what you took away from reading it, then that's more evidence that I didn't do a great job with writing the original draft.

Here's the original paragraph for reference:

Data engineers are the janitors who keep your data clean and flowing. Insights are great, and you need them. But to deliver insights at scale, you need data infrastructure. That’s delivered by data engineering. It’s not as fun to talk about as D3 visualizations and business intelligence dashboards, but it’s every bit as important.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#117

Post by SmellTheGlove » Thu, Sep 08, 2016, 9:11 PM UTC

sloxy wrote: post #94

SmellTheGlove wrote: post #69

Earlier quoted context omitted.

I think it's definitely true. Functionally, I'm a director of data engineering (with a big company, so my real title is way more generic). Usually in the initial screen, we'll talk general dollars, and my number is always out of range. For my level and the fact that I'm reasonably happy where I live now, the number is 200k + relocation (more for Bay Area, but lets not go there), and I don't think that's unreasonable…

"director of data engineering" This is already generic.

Oh boy. My HR title would drive you nuts then. It's just director information management. We also have info analysts and info managers. Our department color is gray.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#118

Post by ISL » Thu, Sep 08, 2016, 9:17 PM UTC

trendia wrote: post #25

snowwrestler wrote: post #21

Earlier quoted context omitted.

This argument comes up all the time on HN, but I don't think it means anything. It seems to me that the ability to fill an opening by offering more salary can't disprove a talent shortage, because it is always possible to do so. Thought experiment: If 100 companies had openings for a skill set that only one person could deliver, all 100 companies could eventually fill their openings by sequentially outbidding each ot…

In this case, though, there are way more than 6,600 people in the US that would be able to get do that data engineering job, including: 1. physicists 2. Wall St. quants 3. game programmers 4. PhD statisticians So, the problem is not that there aren't 6,600 people in the US that can do it, it's that the companies can't pay or don't want to pay the $200,000 + that would be required to hire them.

Or, offer the comparably-compensated part-time job that an academic physicist would accept in parallel with continuing to work in academia.

Source: Am physicist who'd love to find sustainable part-time work at market rates.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#119

Post by ef5a0b0628 » Thu, Sep 08, 2016, 9:20 PM UTC

Every time something comes up on HN about a talent shortage in a field related to software engineering, it hurts. I have been unsuccessfully looking for a full time position since my last start up (I was not a founder) folded six months ago. I have been on over 25 in person interviews and gone through untold degrading whiteboard interviews, code tests, trick questions, and take home projects; all have ended in rejection. This industry has a need to torture candidates because we are all considered to be liars by default. Much is said about combating impostor syndrome in ourselves but we are too eager to engender it in others.

It seems people in this industry refuse to understand that some people are not perfect. I never graduated college because I hated it with the very fiber of my being, so I am not particularly great at white boarding answers to algorithm questions off the top of my head in a high pressure environment. If I need them during my job, I look up answers and learn from people who are much smarter than I am.

My personal identity has been shattered, as I thought my ~5-10 year history of success in the industry indicated I was in demand and talented. I saw posts like this and thought that if the worst happened I'd still be able to find a job. The idea that there is a talent shortage is a lie, or candidates like me wouldn't be treated as I have been. I'm not asking for a free job, or a handout. I have had a successful career so far and am capable of doing good work. But I'm not a specialist in Big Data Machine Learning Neural Networks.

I have struggled with bipolar disorder and suicidal ideation most of my life. I've dealt with the death of my beloved grandmother and my father who was instrumental in my choosing to be an engineer with only minor lapses in control. Nothing has caused me to consider taking my own life as much as the past 6 months. It seems there is no future for me in the only career I have any skill in and which is a huge part of my identity. And to constantly be told that there is such a shortage of engineers only salts the wound.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#120

Post by dizzystar » Thu, Sep 08, 2016, 9:21 PM UTC

mrharrison wrote: post #96

dizzystar wrote: post #55

Earlier quoted context omitted.

As a contradiction to this point, some people (me) really enjoy working with data, from cleaning, munging, creating, sorting, pipelining, etc, and find front-end visualization production excessively boring and mind-numbing. Give me emacs and a command line, and I have all the truth I need, which is far more honest, in my mind, than anything that can be created with D3 or Tableau. Beauty is in the eye of the beholder,…

So I guess you are a data engineer? What makes it fun for you? How do work with your customers to give them what they need in a timely matter? I would be interested to know what stack you use to go from dirty data to customer consumption.

Closer to an aspiring data engineer, though I've done my fair share of ETL, cleaning, database building / rebuilding, admin. Prior jobs have been database engineer, probably closer to DBA.

I just enjoy working with raw data and raw code more than I enjoy writing something that launches a graphic. I enjoy writing a script that finds a bad piece of data, or a script that fixes up everything, or writing something that was once unable to run at all get converted to something that runs in 500ms. Perhaps it is that journey of constant discovery, and seeing that every situation is a unique little puzzle. It is seeing the world as it is with no one reinterpreting what the data means for me. I can explore it and discover what it really means. It is hollow truth, a mess of ideas converted to sets of ideas layered on sets of ideas, and when it is finally drawn down, converted, and passing all tests, it is self-evident and self-reflecting, and true. Hard to explain, but I suppose I like all the things people hate about it.

The tools matter about as much as it matters what CSS framework you are using. You have the ability to logic through UI and UX, whereas I do not. I have zero hope of ever doing well at what you do, since I simply don't have the foundation, but if it matters, I know most jobs I've applied to and worked at tend to be more ad hoc, using PL, Python, Ruby, etc.