Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We’re in the Middle of a Data Engineering Talent Shortage

blog.stitchdata.com

Post reply on HN
Print view View original

151–160 of 166 posts

Re: We’re in the Middle of a Data Engineering Talent Shortage

#151

Post by snovv_crash »

snowwrestler wrote: post #126
snovv_crash wrote: post #63

Earlier quoted context omitted.

Bullshit. I have the skills for an intermediate-level data engineer, but I find it bland and I'd rather work in computer vision. However, offer me enough and I may reconsider, and I don't think I'm alone in this.

You're not alone, but you're just reiterating that it's always possible to fill an opening by running the salary up high enough.

But the pool of people who wouldn't otherwise take the job grows as the salary increases, pulling the people away from careers where they clearly aren't providing as much value to their employers.

Put it this way, the company isn't going to pay the employee more than the value they provide. That is the ceiling on salary. So until that ceiling is reached it is indeed a case of higher bidder takes all, as your thought experiment demonstrated. But once that ceiling is neared the company will make the decision not to bid higher, thus reducing the demand.

Thus, there is no shortage, just a shortage to work at the lower salary of companies with lower ceilings, because they aren't capable of leveraging the employee's talents sufficiently to draw from fields with related skill sets.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#152

Post by Declanomous »

infinite8s wrote: post #146
Declanomous wrote: post #71

Earlier quoted context omitted.

This comment will sound a bit self-serving, but it supports your point. I have most of the skills necessary to be a data engineer. My degree in biology, but I nearly got a double major in computer science with a minor in math. (I wanted to work in bioinformatics, but it's nearly impossible to make more than a pittance without a PhD) I didn't pursue the double major because I felt taking classes outside of those three…

> (I wanted to work in bioinformatics, but it's nearly impossible to make more than a pittance without a PhD) Just wanted to comment on this part - sadly it's difficult to make more than a pittance even with a PhD.

Yeah, totally. The difference is a bioinformaticist with a PhD generally gets to choose what they research, whereas a bioinformaticist without a PhD has to work under someone else's grant. Biology actually has the lowest pay of any major for people working in their field with a four-year degree. You are lucky to make more than minimum wage with a four-year degree, especially if your interest is field ecology or something similar.

If you want to talk about a shortage of labor where it would matter, biology as a field is probably hurting way more for talented software engineers than any company that needs a data engineer. There are so many great applications for programming in biology, and unlike other sciences, say physics, researchers don't tend to pick up on any amount of programming skill on their way to their PhD.

I've tried getting involved in bioinformatics on the side, but it's really difficult to keep up with the field if you don't have thousands of dollars to drop on journal subscriptions. It's also really hard to get access to the data researchers use in general (in any field), but it is made even harder when dealing with researchers involving people due to concerns about privacy. I don't think a focus on privacy is a bad thing, but a lot of publicly available data is sanitized to the point where your sample size would need to be in the billions to draw any inferences. You can request access to less general data, but good luck doing that without the support of a research organization.

Anyways, unless you have a martyr complex, there really isn't any reason to go into bioinformatics.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#153

Post by x0x0 »

Declanomous wrote: post #152
infinite8s wrote: post #146

Earlier quoted context omitted.

> (I wanted to work in bioinformatics, but it's nearly impossible to make more than a pittance without a PhD) Just wanted to comment on this part - sadly it's difficult to make more than a pittance even with a PhD.

Yeah, totally. The difference is a bioinformaticist with a PhD generally gets to choose what they research, whereas a bioinformaticist without a PhD has to work under someone else's grant. Biology actually has the lowest pay of any major for people working in their field with a four-year degree. You are lucky to make more than minimum wage with a four-year degree, especially if your interest is field ecology or somet…

I happily worked in a wetlab writing stats software to support breast cancer research. I now do better ad targeting. My salary tripled.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#154

Post by dredmorbius »

hga wrote: post #143
jballanc wrote: post #133

Earlier quoted context omitted.

Considering that plumbers and janitors have likely, in the entire history of human civilization, done more for health and longevity than doctors and scientists...I'm kind of ok with this analogy.

Doctors, maybe, but it was the scientists who told them about the germ theory of disease, for instance. I've read, but not confirmed for myself, that in the US the biggest gains in health came in the post-Civil War period, when "plumbers and janitors" made the difference. Of course, that's really starting with, after the science, the civil engineers who designed the public works systems that supplied clean water and…

Confirmation: https://ello.co/dredmorbius/post/MsQfdPAn_0XUdZUoReBfbg

Re: We’re in the Middle of a Data Engineering Talent Shortage

#155

Post by JamesBarney »

snowwrestler wrote: post #21
jnordwick wrote: post #11

Whenever I see these posts I immediate translate them in my head to "we're in the middle of a talent shortage at a price I am willing to pay." I've worked with very large amounts of data and high performance computing for most of my career; I mostly had finance related jobs in the last decade or so. I have most of the skill you want, including some you don't know you want. However when salary comes up, that is where…

This argument comes up all the time on HN, but I don't think it means anything. It seems to me that the ability to fill an opening by offering more salary can't disprove a talent shortage, because it is always possible to do so. Thought experiment: If 100 companies had openings for a skill set that only one person could deliver, all 100 companies could eventually fill their openings by sequentially outbidding each ot…

Shortage has a specific term in economics, which pretty much only happens because of price controls.

However if people start liking kale, and the price goes up 20% and you start telling people about the massive kale shortage people will think you're being a little histrionic.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#156

Post by [deleted] »

x0x0 wrote: post #153
Declanomous wrote: post #152

Earlier quoted context omitted.

Yeah, totally. The difference is a bioinformaticist with a PhD generally gets to choose what they research, whereas a bioinformaticist without a PhD has to work under someone else's grant. Biology actually has the lowest pay of any major for people working in their field with a four-year degree. You are lucky to make more than minimum wage with a four-year degree, especially if your interest is field ecology or somet…

I happily worked in a wetlab writing stats software to support breast cancer research. I now do better ad targeting. My salary tripled.

[deleted]

Re: We’re in the Middle of a Data Engineering Talent Shortage

#157

Post by googletazer »

ef5a0b0628 wrote: post #119

Every time something comes up on HN about a talent shortage in a field related to software engineering, it hurts. I have been unsuccessfully looking for a full time position since my last start up (I was not a founder) folded six months ago. I have been on over 25 in person interviews and gone through untold degrading whiteboard interviews, code tests, trick questions, and take home projects; all have ended in reject…

" I have been on over 25 in person interviews and gone through untold degrading whiteboard interviews, code tests, trick questions, and take home projects; all have ended in rejection."

The fact that you pulled through 25 of them is already commendable. Unfortunately as a labor provider you'll be subjected to all kinds of crap for the privilege of working.

Every single person on here needs to have a secondary business going on right now. Doesn't have to be a highly skilled industry either, selling hand made stuff on Etsy can be a lifeline in these situations.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#158

Post by SmellTheGlove »

dizzystar wrote: post #120
mrharrison wrote: post #96

Earlier quoted context omitted.

So I guess you are a data engineer? What makes it fun for you? How do work with your customers to give them what they need in a timely matter? I would be interested to know what stack you use to go from dirty data to customer consumption.

Closer to an aspiring data engineer, though I've done my fair share of ETL, cleaning, database building / rebuilding, admin. Prior jobs have been database engineer, probably closer to DBA. I just enjoy working with raw data and raw code more than I enjoy writing something that launches a graphic. I enjoy writing a script that finds a bad piece of data, or a script that fixes up everything, or writing something that w…

There's nothing aspiring about what you wrote. I think you're fine calling yourself a data engineer if those are the types of challenges you've been solving.

Don't sell yourself short or select yourself out of an opportunity (within reason). That's someone else's job!

Re: We’re in the Middle of a Data Engineering Talent Shortage

#159

Post by SmellTheGlove »

mrharrison wrote: post #138
dizzystar wrote: post #120

Earlier quoted context omitted.

Closer to an aspiring data engineer, though I've done my fair share of ETL, cleaning, database building / rebuilding, admin. Prior jobs have been database engineer, probably closer to DBA. I just enjoy working with raw data and raw code more than I enjoy writing something that launches a graphic. I enjoy writing a script that finds a bad piece of data, or a script that fixes up everything, or writing something that w…

I'm not comparing frontend to backend. I also think data is fun and I don't mean to be little the job, but in a real world scenario its detail intensive, under appreciated, tons of edge cases and extremely complex if you plan to make it scalable and fast. So if you are an aspiring data engineer be aware of these pitfalls, because the first couple times you do it you will think its fun to try something new and create…

You're getting very close to the root cause - customers and even colleagues don't really care about the work that goes into the data. They care about the end deliverable, because that's what creates value for them, and fairly so. That gets at why data engineering as a discipline isn't (IMHO) very well respected.

I know this isn't reddit, so I'll point you to reddit. Check out /r/datascience where those folks talk about what it takes to be a data scientist. Some folks are honest about data engineering, but most handwave past it, or talk about it like it's beneath them. Their role would not be possible without solid data engineering, rather than a complementary and equally important discipline. Good luck doing "data science" or "analytics" or "machine learning" or every other buzzword without clean data, and for us data engineers, good luck ever demonstrating value without the analytics folks working with us.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#160

Post by hga »

dredmorbius wrote: post #154
hga wrote: post #143

Earlier quoted context omitted.

Doctors, maybe, but it was the scientists who told them about the germ theory of disease, for instance. I've read, but not confirmed for myself, that in the US the biggest gains in health came in the post-Civil War period, when "plumbers and janitors" made the difference. Of course, that's really starting with, after the science, the civil engineers who designed the public works systems that supplied clean water and…

Confirmation: https://ello.co/dredmorbius/post/MsQfdPAn_0XUdZUoReBfbg

Thanks! There was a delay after the Civil War as you'd expect from all the chaos and disruption that caused (e.g. MIT got its charter before the outbreak, but wasn't able to start up until after), but it's pretty clear, and gets really dramatic the further you go forward.
Post reply on HN