Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We’re in the Middle of a Data Engineering Talent Shortage

blog.stitchdata.com

Post reply on HN
Print view View original

11–20 of 166 posts

Re: We’re in the Middle of a Data Engineering Talent Shortage

#11

Post by jnordwick »

Whenever I see these posts I immediate translate them in my head to "we're in the middle of a talent shortage at a price I am willing to pay."

I've worked with very large amounts of data and high performance computing for most of my career; I mostly had finance related jobs in the last decade or so. I have most of the skill you want, including some you don't know you want. However when salary comes up, that is where we start to part ways. If you are really serious about a shortage, you should be really serious about making offers that can be competitive, but I keep seeing the same $150k offers. That isn't a "shortage" kind of offer.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#13

Post by pjmlp »

PaulHoule wrote: post #5
mattnewton wrote: post #2

I'm trying to switch careers into "Data Engineering" now, as a full stack developer who is more interested in ML, and I've found almost no traction internally at my company or externally. It looks like I may just accept a full stack position at a good company that does a lot of data science for now, but though I would ask - Where are all these jobs?

"Data Engineering" is most of the work that needs to be done, but I think companies haven't identified it as a category. From my P.O.V., "Full Stack Engineer" is a place you don't want to be because it means putting out fires with whatever junk javascript is in the front end. It seems like everybody who's built a serious javascript application has invented their own Virtual DOM because none of the popular Virtual DOM…

There are "Full Stack Engineer" doing pure native applications, which is what I have been doing the last three years after escaping the web back into native land.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#14

Post by tobyjsullivan »

jnordwick wrote: post #11

Whenever I see these posts I immediate translate them in my head to "we're in the middle of a talent shortage at a price I am willing to pay." I've worked with very large amounts of data and high performance computing for most of my career; I mostly had finance related jobs in the last decade or so. I have most of the skill you want, including some you don't know you want. However when salary comes up, that is where…

"we're in the middle of a talent shortage [and don't believe in upskilling]."

Re: We’re in the Middle of a Data Engineering Talent Shortage

#15

Post by GeneralMayhem »

There are 6600 jobs listed and 6500 individuals on LinkedIn with that particular title, and therefore there's a shortage? Seriously?

* How many aren't on LinkedIn?

* Since the whole article is about how the job title is poorly defined and growing in prevalence, why would you assume that people who don't already have such a job would use the term?

* The "growth" charts on the full study are just as bad - how much of that is just from renaming existing generic developer positions, since "data engineer" is clearly a relatively new term?

Re: We’re in the Middle of a Data Engineering Talent Shortage

#17

Post by Swizec »

skynetv2 wrote: post #3

anything and everything is marketed as "data science" and "data engineering" these days becasue this is the buzzword of the day. I've been dealing with large data even before "big data" was a word but i dont call myself "data scientist" or "data engineer". I am still a software engineer working on what benefits my organization. "Serial Entrepreneur" is the same these days, claimed by anyone who had a lemonade stand a…

> I am still a software engineer working on what benefits my organization

But if you saw a nearby local maximum that's higher than your current local maximum, wouldn't you change what you call yourself, if it means being paid more but doing the same work?

This is similar to how the average "software engineer" makes about $30k/year more than the average "programmer".

Re: We’re in the Middle of a Data Engineering Talent Shortage

#18

Post by ThomPete »

tobyjsullivan wrote: post #14
jnordwick wrote: post #11

Whenever I see these posts I immediate translate them in my head to "we're in the middle of a talent shortage at a price I am willing to pay." I've worked with very large amounts of data and high performance computing for most of my career; I mostly had finance related jobs in the last decade or so. I have most of the skill you want, including some you don't know you want. However when salary comes up, that is where…

"we're in the middle of a talent shortage [and don't believe in upskilling]."

Upskilling is one of the most ineffective costly ways to try and "re-program" workers and it mostly doesn't work because it's not about skills it's about talent.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#19

Post by kafkaesq »

dmatthewson wrote: post #16

From the article: "Data engineers are the janitors who keep your data clean and flowing." Hm, I wonder why he's having problems hiring janitors.

Data engineers are the janitors who keep your data clean and flowing.

In a boldface font, no less. The cockiness behind that language is really quite astounding.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#20

Post by achompas »

mattnewton wrote: post #2

I'm trying to switch careers into "Data Engineering" now, as a full stack developer who is more interested in ML, and I've found almost no traction internally at my company or externally. It looks like I may just accept a full stack position at a good company that does a lot of data science for now, but though I would ask - Where are all these jobs?

My official title is "Data Scientist" although I'm closer to the "ML Engineer" someone else mentions in a child comment.

Frankly speaking, if your company doesn't need a data engineer, it won't hire one or move you into that role. They likely don't, either, if you're experiencing this pushback -- data engineers often develop ETL pipelines or data warehouses, both of which are very useful if your company has a data team and very useless if it does not.

That said, you may want to move closer to my role. There's actually a shortage of data-savvy people who can also write production software, and you would nicely complement a more research-inclined data scientist or analyst -- someone with far more experience with research/analysis than development.

Post reply on HN