We’re in the Middle of a Data Engineering Talent Shortage • Hacker News Board

Re: We’re in the Middle of a Data Engineering Talent Shortage

#161

Post by Grishnakh » Fri, Sep 09, 2016, 9:03 PM UTC

Declanomous wrote: post #71

trendia wrote: post #25

Earlier quoted context omitted.

In this case, though, there are way more than 6,600 people in the US that would be able to get do that data engineering job, including: 1. physicists 2. Wall St. quants 3. game programmers 4. PhD statisticians So, the problem is not that there aren't 6,600 people in the US that can do it, it's that the companies can't pay or don't want to pay the $200,000 + that would be required to hire them.

This comment will sound a bit self-serving, but it supports your point. I have most of the skills necessary to be a data engineer. My degree in biology, but I nearly got a double major in computer science with a minor in math. (I wanted to work in bioinformatics, but it's nearly impossible to make more than a pittance without a PhD) I didn't pursue the double major because I felt taking classes outside of those three…

>The work I do makes a difference. Not in a "I make something people use" difference,...

I'd just be happy with that. Most of the work I've done professionally hasn't gone anywhere; it's always "we missed the market window" or "upper management decided on a new strategy". I can't point to that many things I got to work on that actually made it into the market and were used by people for long. One place (a semiconductor company) had a successful though buggy product and large customers in place, with the product already deployed into the field, and the software I wrote got used by some customers, but then suddenly the company decided they weren't making a big enough profit margin on this part (even though the profits were guaranteed and extremely low-risk as the customers had the part designed-in), so they simply quit the market and laid off our entire team.

Making something people use would be a step up. Rescuing people and feeding starving kids is a pipe dream, but then again I work on embedded devices, not big data or analytics or anything like that so that's not exactly a position that'd be easy for me to find if I really wanted it.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#162

Post by Grishnakh » Fri, Sep 09, 2016, 9:14 PM UTC

wavefunction wrote: post #86

kafkaesq wrote: post #76

Earlier quoted context omitted.

So is the work that doctors, lawyers, and other highly-skilled people do, by and large. Everyone knows that day-to-day aspects of these jobs are hardly glamorous (or even cerebral), the vast majority of the time. Yet somehow we manage to accord these people with their due degree of respect, and wouldn't think of referring to them as "janitors".

I don't see what's so bad about janitors, though. They do very thankless jobs for not much money, whereas doctors and lawyers and other high-skilled individuals are often well renumerated or offered certain social prestige that your post shows is quite lacking when a humble janitor is considered.

There's nothing wrong with janitors, but the work they do can be done by any dumb monkey with almost no training. That's why the pay is so low for the job: anyone can do it, as long as they can lift trash cans and push a vacuum cleaner.

Plumbers are entirely different. They have to get their hands dirty working on some awful systems, but they actually have to know what they're doing, get specialized training, etc. Soldering a proper joint with copper pipes isn't that easy, and if you screw it up, it'll leak later and cause a lot of property damage. Knowing which pipes and fittings to use where is specialized knowledge. It's not something you can just grab someone off the street and train them to do in 30 minutes. Of course, plumbers also cost a lot too, and the ones who are self-employed (rather than their assistants) generally do pretty well financially.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#163

Post by dredmorbius » Fri, Sep 09, 2016, 10:13 PM UTC

hga wrote: post #160

dredmorbius wrote: post #154

Earlier quoted context omitted.

Confirmation: https://ello.co/dredmorbius/post/MsQfdPAn_0XUdZUoReBfbg

Thanks! There was a delay after the Civil War as you'd expect from all the chaos and disruption that caused (e.g. MIT got its charter before the outbreak, but wasn't able to start up until after), but it's pretty clear, and gets really dramatic the further you go forward.

I've been studying the period (mostly the Industrial Revolution and onward, though the accelleration of the late 19th / early 20th century is staggering), and it's pretty phenomenal.

There was a lot going on. Germ theory, of course, was part of it. But public health measures, especially sewerage systems, clean drinking water, and municipal waste removal, were all massive contributors. Note that the decline in mortality occurs well in advance of antibiotics and even most vaccinations.

For all the recent debate on vaccinations, it's interesting to note that the peak period of their impace (roughly 1930 - 1960) saw relatively little reduction in mortality, though there was a large decrease in disease incidence. It turns out that with septic control, antibiotics, food quality, and nutrition, many viral diseases weren't killers, but did present quality-of-life issues. And yes, often quite severe -- polio was no joke, and I know people who've suffered lameness from it myself. Measles and smallpox are similarly scarring and have long-term impacts.

But the major impacts of virtually all medicine are front-loaded to the period before 1950, with much the gains since attributable to either greater access (especially for the disadvantaged) and removal of environmental agonists (lead, tobacco, alcohol, asbestos, miscellaneous poisons, safety hazards).

Re: We’re in the Middle of a Data Engineering Talent Shortage

#164

Post by jballanc » Fri, Sep 09, 2016, 10:21 PM UTC

hga wrote: post #143

jballanc wrote: post #133

Earlier quoted context omitted.

Considering that plumbers and janitors have likely, in the entire history of human civilization, done more for health and longevity than doctors and scientists...I'm kind of ok with this analogy.

Doctors, maybe, but it was the scientists who told them about the germ theory of disease, for instance. I've read, but not confirmed for myself, that in the US the biggest gains in health came in the post-Civil War period, when "plumbers and janitors" made the difference. Of course, that's really starting with, after the science, the civil engineers who designed the public works systems that supplied clean water and…

Sure, that's true of recent (< 200 yr ago) history, but plumbing's contribution to health and longevity goes all the way back to ancient Rome. (Somewhat ironically, plumbing, from the Latin word for lead, "plumbus", may have also contributed indirectly to Rome's eventual decline.)

Re: We’re in the Middle of a Data Engineering Talent Shortage

#165

Post by xapata » Fri, Sep 09, 2016, 11:22 PM UTC

SmellTheGlove wrote: post #128

biztos wrote: post #113

Earlier quoted context omitted.

Maybe you should start a Data Science & Engineering consultancy. The same people who would offer $150K to an employee often have bosses who would love to spend $500K for a person-year of (contract) work if it comes with a high probability of success.

I have thought about that. Many times. I have a couple of barriers, most of which are temporary: 1. I have student debt from my law degree, and I have a lower risk tolerance until that's paid off. 2. My daughter is 4, it's nice to be around for the early years, and the corporate gig is quite comfortable in terms of hours. 3. I'm in Maine. Most clients would require me to travel, which impacts #2. I do have a former c…

Running an individual or small consulting company isn't that hard. Scaling a consulting company, on the other hand, is quite difficult.

Re: We’re in the Middle of a Data Engineering Talent Shortage

#166

Post by banku_brougham » Sat, Sep 10, 2016, 5:57 AM UTC

ajamesm wrote: post #122

banku_brougham wrote: post #85

Earlier quoted context omitted.

sed -i 's/emacs/sublime-text/g' what_u_said.txt

more like Ctrl-H, tab, 'emacs', tab, 'sublime-text', tab, enter, esc, Ctrl-S

you are right that is more coherent.