We’re in the Middle of a Data Engineering Talent Shortage • Hacker News Board

Re: We’re in the Middle of a Data Engineering Talent Shortage

#51

Post by p4wnc6 » Thu, Sep 08, 2016, 7:14 PM UTC

beachstartup wrote: post #37

tuna-piano wrote: post #30

Earlier quoted context omitted.

Of course this is more or less always true - there are only shortages or excesses of things when prices don't or can't adjust freely. If there was 1 gallon of water left on earth, Bill gates would buy that gallon for $50 billion, and everyone else would die of dehydration. There has always been a shortage of maids willing to do all my house work for $10. And there is a shortage of data engineers at $x, but there woul…

> If there was 1 gallon of water left on earth, Bill gates would buy that gallon for $50 billion, and everyone else would die of dehydration. really? who would sell the last gallon of water on earth?

Disney would.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#52

Post by rhizome » Thu, Sep 08, 2016, 7:15 PM UTC

makmanalp wrote: post #46

Quick sidenote, anyone know where the databases / distributed systems engineering jobs are at? E.g. if one wanted to not use these tools but also go help build these tools? I can think of Facebook, Google, Microsoft, IBM (which locations and groups within these companies / where?). I can also think of Confluent, CitusDB, Databricks, etc.

Market Research is a $40B industry that depends almost completely on these concepts. I'm not sure how prevalent distributed systems are with MR companies, but that's an implementation detail anyway.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#53

Post by protomyth » Thu, Sep 08, 2016, 7:19 PM UTC

I worked for about 10 years doing exactly what they want, but I ended up having to write a lot of the tools which means I'm not able to check the boxes on some tool you require which gets me punted by HR.

I'm starting to think that the message is if HR is going to do checklists then developers should really make sure they work mostly with contracts that use popular checklist items.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#54

Post by praccu » Thu, Sep 08, 2016, 7:20 PM UTC

kafkaesq wrote: post #19

dmatthewson wrote: post #16

From the article: "Data engineers are the janitors who keep your data clean and flowing." Hm, I wonder why he's having problems hiring janitors.

Data engineers are the janitors who keep your data clean and flowing. In a boldface font, no less. The cockiness behind that language is really quite astounding.

It's really true, though. It's brutal, ugly work with no hope of an end.

Edit: Favorite paper on the topic: http://research.google.com/pubs/pub43146.html

Re: We’re in the Middle of a Data Engineering Talent Shortage

#55

Post by dizzystar » Thu, Sep 08, 2016, 7:21 PM UTC

mrharrison wrote: post #27

We should rename this job position to Data Sanity Engineers. I have been thrown these projects at work before, where I'm the frontend engineer and I need to make some cool D3 visualization, but low behold the data is shit, and I have to help the backend team make the data useable. It's a mind-numbing job, that nobody wants, because it sounds like a one month task to get a good REST API up and working, but it usually…

As a contradiction to this point, some people (me) really enjoy working with data, from cleaning, munging, creating, sorting, pipelining, etc, and find front-end visualization production excessively boring and mind-numbing.

Give me emacs and a command line, and I have all the truth I need, which is far more honest, in my mind, than anything that can be created with D3 or Tableau. Beauty is in the eye of the beholder, and it doesn't really do anyone service to look down on the work others find enjoyable. If doing D3 makes you happy, that is awesome, and I can only congratulate you for your passion and your ability to look forward to work I don't "get," and I wish the feelings would be mutual.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#56

Post by serge2k » Thu, Sep 08, 2016, 7:22 PM UTC

rhizome wrote: post #52

makmanalp wrote: post #46

Quick sidenote, anyone know where the databases / distributed systems engineering jobs are at? E.g. if one wanted to not use these tools but also go help build these tools? I can think of Facebook, Google, Microsoft, IBM (which locations and groups within these companies / where?). I can also think of Confluent, CitusDB, Databricks, etc.

Market Research is a $40B industry that depends almost completely on these concepts. I'm not sure how prevalent distributed systems are with MR companies, but that's an implementation detail anyway.

> that's an implementation detail anyway.

Which is what the poster was asking for.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#57

Post by atombath » Thu, Sep 08, 2016, 7:22 PM UTC

qaq wrote: post #49

spoonie wrote: post #39

Earlier quoted context omitted.

Someone who just drank the last 2nd last gallon. ;)

and what would he use the money for :)?

bullets =)

Re: We’re in the Middle of a Data Engineering Talent Shortage

#58

Post by SmellTheGlove » Thu, Sep 08, 2016, 7:23 PM UTC

mrharrison wrote: post #27

We should rename this job position to Data Sanity Engineers. I have been thrown these projects at work before, where I'm the frontend engineer and I need to make some cool D3 visualization, but low behold the data is shit, and I have to help the backend team make the data useable. It's a mind-numbing job, that nobody wants, because it sounds like a one month task to get a good REST API up and working, but it usually…

I do it for a living. It seems underappreciated in the industry.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#59

Post by jakestein » Thu, Sep 08, 2016, 7:23 PM UTC

dmatthewson wrote: post #16

From the article: "Data engineers are the janitors who keep your data clean and flowing." Hm, I wonder why he's having problems hiring janitors.

Hey I'm the author of this blog post and the CEO of the company that did the benchmark report. That was a very poor choice of words on my part, and I appreciate you flagging it. I reworked the paragraph to remove the janitor comment and (hopefully) make it clearer.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#60

Post by [deleted] » Thu, Sep 08, 2016, 7:25 PM UTC

tuna-piano wrote: post #30

jnordwick wrote: post #11

Whenever I see these posts I immediate translate them in my head to "we're in the middle of a talent shortage at a price I am willing to pay." I've worked with very large amounts of data and high performance computing for most of my career; I mostly had finance related jobs in the last decade or so. I have most of the skill you want, including some you don't know you want. However when salary comes up, that is where…

Of course this is more or less always true - there are only shortages or excesses of things when prices don't or can't adjust freely. If there was 1 gallon of water left on earth, Bill gates would buy that gallon for $50 billion, and everyone else would die of dehydration. There has always been a shortage of maids willing to do all my house work for $10. And there is a shortage of data engineers at $x, but there woul…

[deleted]