Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We’re in the Middle of a Data Engineering Talent Shortage

blog.stitchdata.com

Post reply on HN
Print view View original

81–90 of 166 posts

Re: We’re in the Middle of a Data Engineering Talent Shortage

#81

Post by crispyambulance »

ThomPete wrote: post #18
tobyjsullivan wrote: post #14

Earlier quoted context omitted.

"we're in the middle of a talent shortage [and don't believe in upskilling]."

Upskilling is one of the most ineffective costly ways to try and "re-program" workers and it mostly doesn't work because it's not about skills it's about talent.

I sort of agree in that orgs can't simply create massive education programs to re-purpose skill-sets/talent. That might have been possible "back in the day" before project managers were breathing down people's necks, but not today.

But the brightside is that talented people will find a way to "upskill" themselves in whatever environment they find themselves in. It is then up to the candidates to sell themselves and for the potential employers to be flexible about considering different backgrounds and nurturing the development of cross-functional skills that are needed for so-called data-engineers.

The skills listed in the article are all fairly common but its hard to find enough of these skills within individuals. For example, its not hard to find folks who can do the care and feeding of sql-server databases, or skilled programmers, or analysts who understand the business domain intimately. The problem is getting all of these together in one individual in a "know-enough-to-be-dangerous" level.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#82

Post by sbov »

stale2002 wrote: post #68
jnordwick wrote: post #11

Whenever I see these posts I immediate translate them in my head to "we're in the middle of a talent shortage at a price I am willing to pay." I've worked with very large amounts of data and high performance computing for most of my career; I mostly had finance related jobs in the last decade or so. I have most of the skill you want, including some you don't know you want. However when salary comes up, that is where…

Although your statement is technically true, it is basically meaningless. Yes, you can always always always find somebody to do a job is your a willing to pay 10 million dollars. That means that "shortages" are impossible. It means that you can never have a shortage in any situation, because you can always pay 10 million dollars for a single visit to the doctor. But this line of logic isn't very useful when talking a…

Haven't you just done a 180? I mean, I'm pretty sure the world would be in a much much better place if the government could snap its fingers and instantly produce large amounts of almost anything. Therefore there is a shortage of almost everything.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#83

Post by harichinnan »

jnordwick wrote: post #11

Whenever I see these posts I immediate translate them in my head to "we're in the middle of a talent shortage at a price I am willing to pay." I've worked with very large amounts of data and high performance computing for most of my career; I mostly had finance related jobs in the last decade or so. I have most of the skill you want, including some you don't know you want. However when salary comes up, that is where…

We all like $400K the investment bankers make. But Finance Industry had developed a business where they could pay their workers $400K and still make a huge profit for their investors. Except for Googles and Facebooks, the average tech startup is not making Finance industry level profits.

Also Finance requires proper education and training. Not so much for App development. So for everyone who complains about getting $150K offers, there are a 100 thousand people right here in US applying for $60K technical analyst jobs.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#84

Post by wavefunction »

bobosha wrote: post #62
jnordwick wrote: post #11

Whenever I see these posts I immediate translate them in my head to "we're in the middle of a talent shortage at a price I am willing to pay." I've worked with very large amounts of data and high performance computing for most of my career; I mostly had finance related jobs in the last decade or so. I have most of the skill you want, including some you don't know you want. However when salary comes up, that is where…

>Whenever I see these posts I immediate translate them in my head to "we're in the middle of a talent shortage at a price I am willing to pay." That's true for just about anything. "there is no epipen crisis, only a crisis at what you are willing to pay" "There is no poverty , only poverty at a given income level" *"there is no crime problem, only crime problem at a given crime level" what you are saying is self-cont…

You must admit that the price of epipens is an artificially inflated one only possible due to government imposed monopoly, not one driving by true market forces.

Poverty is simply a description wealth and is always comparative. We can define poverty as any level we so desire.

One might argue that any crime is a problem, as long as it causes an issue for society or victims.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#85

Post by banku_brougham »

dizzystar wrote: post #55
mrharrison wrote: post #27

We should rename this job position to Data Sanity Engineers. I have been thrown these projects at work before, where I'm the frontend engineer and I need to make some cool D3 visualization, but low behold the data is shit, and I have to help the backend team make the data useable. It's a mind-numbing job, that nobody wants, because it sounds like a one month task to get a good REST API up and working, but it usually…

As a contradiction to this point, some people (me) really enjoy working with data, from cleaning, munging, creating, sorting, pipelining, etc, and find front-end visualization production excessively boring and mind-numbing. Give me emacs and a command line, and I have all the truth I need, which is far more honest, in my mind, than anything that can be created with D3 or Tableau. Beauty is in the eye of the beholder,…

    sed -i 's/emacs/sublime-text/g' what_u_said.txt

Re: We’re in the Middle of a Data Engineering Talent Shortage

#86

Post by wavefunction »

kafkaesq wrote: post #76
praccu wrote: post #54

Earlier quoted context omitted.

It's really true, though. It's brutal, ugly work with no hope of an end. Edit: Favorite paper on the topic: http://research.google.com/pubs/pub43146.html

So is the work that doctors, lawyers, and other highly-skilled people do, by and large. Everyone knows that day-to-day aspects of these jobs are hardly glamorous (or even cerebral), the vast majority of the time. Yet somehow we manage to accord these people with their due degree of respect, and wouldn't think of referring to them as "janitors".

I don't see what's so bad about janitors, though. They do very thankless jobs for not much money, whereas doctors and lawyers and other high-skilled individuals are often well renumerated or offered certain social prestige that your post shows is quite lacking when a humble janitor is considered.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#87

Post by wavefunction »

jakestein wrote: post #59
dmatthewson wrote: post #16

From the article: "Data engineers are the janitors who keep your data clean and flowing." Hm, I wonder why he's having problems hiring janitors.

Hey I'm the author of this blog post and the CEO of the company that did the benchmark report. That was a very poor choice of words on my part, and I appreciate you flagging it. I reworked the paragraph to remove the janitor comment and (hopefully) make it clearer.

You should also not use "janitor" as a disparaging term. That would be another good takeaway from all of this.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#88

Post by cheriot »

It was only 20 years ago that companies hired a "web master" or a generalist to do everything. But pieces of those jobs became specialized. Now we need UX, UI programmer, general engineers, dev ops, data engineers, a data scientist, etc.

And how many companies are still interviewing with fizzbuzz?

Re: We’re in the Middle of a Data Engineering Talent Shortage

#89

Post by SmellTheGlove »

dizzystar wrote: post #78
jnordwick wrote: post #11

Whenever I see these posts I immediate translate them in my head to "we're in the middle of a talent shortage at a price I am willing to pay." I've worked with very large amounts of data and high performance computing for most of my career; I mostly had finance related jobs in the last decade or so. I have most of the skill you want, including some you don't know you want. However when salary comes up, that is where…

Are they looking for someone who must have every box ticked or are they looking for someone with enough qualifications yet needing work so much they are willing to undercut themselves? Are they justifying their salary offer because you tick 90% of the boxes and not 100%? I've been looking for work in data engineering and databases for 9 months, and while I'm certainly not as qualified and experienced as you are, I co…

That to me is a classic recruiting problem in technical positions, data engineering included. Unless you have a manager handling it themselves, the person doing the initial screen really is ticking boxes because they may not know any better.

Once a resume gets to me, and I'm only speaking for myself here, I'm looking for the challenges you've faced and the problems you've solved. I actually care very little about what tech you used because odds are we'll have something different, but we'll need to solve problems. If someone is solid in some related technical skillset, can think critically, and communicate the details of what they've tackled in the past, learning our specific tech stack is going to be the easy part.

Let me put it another way - when I look for interns or entry level hires, the number of those that can do more than spell SAS or Teradata approaches zero very quickly. But if they've solved challenges of the magnitude that they'd be expected to solve with us initially, the tech is secondary to process and problem solving. As we look more experienced, I'd still be limiting myself to candidates from a set of "legacy" industries that prefer these sorts of tools if I insisted on checking those boxes at the outset. I'd prefer to teach a really smart person to use the things that they don't know yet if I have it my way.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#90

Post by triplesec »

p4wnc6 wrote: post #40
pavlov wrote: post #29

Earlier quoted context omitted.

Bizarrely, I remember a recent HN discussion where a poster was arguing that any software developer who is not working in machine learning is like a plumber. I guess this means that the entire profession consists of janitors and plumbers.

In the minds of middle management, I think this is precisely correct.

and as pointed out so much, is entirely why nobody wants to work for them. Respect these very bright people znd you have a starting negotiation position.
Post reply on HN