We’re in the Middle of a Data Engineering Talent Shortage

Post by mattnewton » Thu, Sep 08, 2016, 6:34 PM UTC

alexbeloi wrote: post #9

mattnewton wrote: post #2

I'm trying to switch careers into "Data Engineering" now, as a full stack developer who is more interested in ML, and I've found almost no traction internally at my company or externally. It looks like I may just accept a full stack position at a good company that does a lot of data science for now, but though I would ask - Where are all these jobs?

You might want to look at "Machine Learning Engineer" positions if you want to do ML in practice, it's starting to be a title I see somewhat often now. As others have pointed out Data Engineering is more about building data pipelines, making architecture decisions for your ML stack, things like that. Less about model building, prototyping and training, which is what I think of when somebody says they 'do' ML.

Post by [deleted] » Thu, Sep 08, 2016, 6:37 PM UTC

snowwrestler wrote: post #21

jnordwick wrote: post #11

Whenever I see these posts I immediate translate them in my head to "we're in the middle of a talent shortage at a price I am willing to pay." I've worked with very large amounts of data and high performance computing for most of my career; I mostly had finance related jobs in the last decade or so. I have most of the skill you want, including some you don't know you want. However when salary comes up, that is where…

This argument comes up all the time on HN, but I don't think it means anything. It seems to me that the ability to fill an opening by offering more salary can't disprove a talent shortage, because it is always possible to do so. Thought experiment: If 100 companies had openings for a skill set that only one person could deliver, all 100 companies could eventually fill their openings by sequentially outbidding each ot…

Post by linkregister » Thu, Sep 08, 2016, 6:40 PM UTC

ThomPete wrote: post #18

tobyjsullivan wrote: post #14

Earlier quoted context omitted.

"we're in the middle of a talent shortage [and don't believe in upskilling]."

Upskilling is one of the most ineffective costly ways to try and "re-program" workers and it mostly doesn't work because it's not about skills it's about talent.

Post by beachstartup » Thu, Sep 08, 2016, 6:45 PM UTC

tuna-piano wrote: post #30

jnordwick wrote: post #11

Whenever I see these posts I immediate translate them in my head to "we're in the middle of a talent shortage at a price I am willing to pay." I've worked with very large amounts of data and high performance computing for most of my career; I mostly had finance related jobs in the last decade or so. I have most of the skill you want, including some you don't know you want. However when salary comes up, that is where…

Of course this is more or less always true - there are only shortages or excesses of things when prices don't or can't adjust freely. If there was 1 gallon of water left on earth, Bill gates would buy that gallon for $50 billion, and everyone else would die of dehydration. There has always been a shortage of maids willing to do all my house work for $10. And there is a shortage of data engineers at $x, but there woul…

Post by spoonie » Thu, Sep 08, 2016, 6:55 PM UTC

beachstartup wrote: post #37

tuna-piano wrote: post #30

Earlier quoted context omitted.

Of course this is more or less always true - there are only shortages or excesses of things when prices don't or can't adjust freely. If there was 1 gallon of water left on earth, Bill gates would buy that gallon for $50 billion, and everyone else would die of dehydration. There has always been a shortage of maids willing to do all my house work for $10. And there is a shortage of data engineers at $x, but there woul…

> If there was 1 gallon of water left on earth, Bill gates would buy that gallon for $50 billion, and everyone else would die of dehydration. really? who would sell the last gallon of water on earth?