Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We’re in the Middle of a Data Engineering Talent Shortage

blog.stitchdata.com

Post reply on HN
Print view View original

121–130 of 166 posts

Re: We’re in the Middle of a Data Engineering Talent Shortage

#121

Post by mrharrison »

SmellTheGlove wrote: post #58
mrharrison wrote: post #27

We should rename this job position to Data Sanity Engineers. I have been thrown these projects at work before, where I'm the frontend engineer and I need to make some cool D3 visualization, but low behold the data is shit, and I have to help the backend team make the data useable. It's a mind-numbing job, that nobody wants, because it sounds like a one month task to get a good REST API up and working, but it usually…

I do it for a living. It seems underappreciated in the industry.

I agree. I enjoyed doing it the first couple times, but people would often complain why I wasn't done sooner and didn't appreciate the level of complexity that went in to doing it. Once the appreciation was gone, I believe that's when it turned into a mind-numbing task for me. I don't mean to be little the job, I think I have just become sour to it because of the lack of appreciation.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#122

Post by ajamesm »

banku_brougham wrote: post #85
dizzystar wrote: post #55

Earlier quoted context omitted.

As a contradiction to this point, some people (me) really enjoy working with data, from cleaning, munging, creating, sorting, pipelining, etc, and find front-end visualization production excessively boring and mind-numbing. Give me emacs and a command line, and I have all the truth I need, which is far more honest, in my mind, than anything that can be created with D3 or Tableau. Beauty is in the eye of the beholder,…

sed -i 's/emacs/sublime-text/g' what_u_said.txt

more like Ctrl-H, tab, 'emacs', tab, 'sublime-text', tab, enter, esc, Ctrl-S

Re: We’re in the Middle of a Data Engineering Talent Shortage

#123

Post by FLUX-YOU »

achompas wrote: post #20
mattnewton wrote: post #2

I'm trying to switch careers into "Data Engineering" now, as a full stack developer who is more interested in ML, and I've found almost no traction internally at my company or externally. It looks like I may just accept a full stack position at a good company that does a lot of data science for now, but though I would ask - Where are all these jobs?

My official title is "Data Scientist" although I'm closer to the "ML Engineer" someone else mentions in a child comment. Frankly speaking, if your company doesn't need a data engineer, it won't hire one or move you into that role. They likely don't, either, if you're experiencing this pushback -- data engineers often develop ETL pipelines or data warehouses, both of which are very useful if your company has a data te…

>There's actually a shortage of data-savvy people who can also write production software

Well no kidding, that's one person doing two jobs. That's easily a 5-10 year training time depending on how high a quality you demand from their production software.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#124

Post by __derek__ »

ISL wrote: post #118
trendia wrote: post #25

Earlier quoted context omitted.

In this case, though, there are way more than 6,600 people in the US that would be able to get do that data engineering job, including: 1. physicists 2. Wall St. quants 3. game programmers 4. PhD statisticians So, the problem is not that there aren't 6,600 people in the US that can do it, it's that the companies can't pay or don't want to pay the $200,000 + that would be required to hire them.

Or, offer the comparably-compensated part-time job that an academic physicist would accept in parallel with continuing to work in academia. Source: Am physicist who'd love to find sustainable part-time work at market rates.

This is on-point. For comparison's sake, look at the number of economists, political scientists, and business professors who have side gigs in consulting.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#125

Post by __derek__ »

harichinnan wrote: post #83
jnordwick wrote: post #11

Whenever I see these posts I immediate translate them in my head to "we're in the middle of a talent shortage at a price I am willing to pay." I've worked with very large amounts of data and high performance computing for most of my career; I mostly had finance related jobs in the last decade or so. I have most of the skill you want, including some you don't know you want. However when salary comes up, that is where…

We all like $400K the investment bankers make. But Finance Industry had developed a business where they could pay their workers $400K and still make a huge profit for their investors. Except for Googles and Facebooks, the average tech startup is not making Finance industry level profits. Also Finance requires proper education and training. Not so much for App development. So for everyone who complains about getting $…

> Except for Googles and Facebooks, the average tech startup is not making Finance industry level profits.

And they don't have finance/Google/Facebook level needs for data engineers. They can't reasonably claim to need top-level skills and then beggar out on the cost.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#126

Post by snowwrestler »

snovv_crash wrote: post #63
snowwrestler wrote: post #21

Earlier quoted context omitted.

This argument comes up all the time on HN, but I don't think it means anything. It seems to me that the ability to fill an opening by offering more salary can't disprove a talent shortage, because it is always possible to do so. Thought experiment: If 100 companies had openings for a skill set that only one person could deliver, all 100 companies could eventually fill their openings by sequentially outbidding each ot…

Bullshit. I have the skills for an intermediate-level data engineer, but I find it bland and I'd rather work in computer vision. However, offer me enough and I may reconsider, and I don't think I'm alone in this.

You're not alone, but you're just reiterating that it's always possible to fill an opening by running the salary up high enough.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#127

Post by x0x0 »

jnordwick wrote: post #11

Whenever I see these posts I immediate translate them in my head to "we're in the middle of a talent shortage at a price I am willing to pay." I've worked with very large amounts of data and high performance computing for most of my career; I mostly had finance related jobs in the last decade or so. I have most of the skill you want, including some you don't know you want. However when salary comes up, that is where…

In particular, employers whining about lack of X need to ponder raising wages to where employees can afford homes in a city where prices are now within spitting distance of $1k/ft2. When your basic pitch is, "We desperately need [data engineers | machine learning engineers | computer vision engineers | what have you] so desperate to live in CA they'll accept never being able to afford a home unless our lottery tickets pay out", it should be unsurprising they have a hard time finding the talent they claim to need. Or, they could accept remote workers! Even remote workers near sfbay, who just don't want to burn 2.5 hours/day commuting in and out of sf...

Re: We’re in the Middle of a Data Engineering Talent Shortage

#128

Post by SmellTheGlove »

biztos wrote: post #113
jnordwick wrote: post #11

Whenever I see these posts I immediate translate them in my head to "we're in the middle of a talent shortage at a price I am willing to pay." I've worked with very large amounts of data and high performance computing for most of my career; I mostly had finance related jobs in the last decade or so. I have most of the skill you want, including some you don't know you want. However when salary comes up, that is where…

Maybe you should start a Data Science & Engineering consultancy. The same people who would offer $150K to an employee often have bosses who would love to spend $500K for a person-year of (contract) work if it comes with a high probability of success.

I have thought about that. Many times. I have a couple of barriers, most of which are temporary:

1. I have student debt from my law degree, and I have a lower risk tolerance until that's paid off. 2. My daughter is 4, it's nice to be around for the early years, and the corporate gig is quite comfortable in terms of hours. 3. I'm in Maine. Most clients would require me to travel, which impacts #2.

I do have a former colleague here that started a data consultancy. I should grab a beer with him and see if we have common ground in the short term. It's not quite starting your own thing, but it might be fun.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#129

Post by slantedview »

These "shortage" stories always make me roll my eyes, because they're usually about money more than anything. And money is usually about cost of living more than anything.

If you choose to locate your company in one of the highest cost of living regions in the world, then you are complicit in the "shortage". Supply and demand - pay up. Or don't.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#130

Post by bcbrown »

mattnewton wrote: post #2

I'm trying to switch careers into "Data Engineering" now, as a full stack developer who is more interested in ML, and I've found almost no traction internally at my company or externally. It looks like I may just accept a full stack position at a good company that does a lot of data science for now, but though I would ask - Where are all these jobs?

You should put your email in your profile. If you're in Seattle, send me an email.
Post reply on HN