Big Data's Big Problem: Little Talent • Hacker News Board

Re: Big Data's Big Problem: Little Talent

#61

Post by yummyfajitas » Sat, Apr 28, 2012, 4:12 PM UTC

Tichy wrote: post #21

yummyfajitas wrote: post #19

Earlier quoted context omitted.

Companies already do pay close to $200k/year for entry level data scientists. (what other kind of scientists are there? The tea-leaf reading kind?) "Data scientist" refers to the guy who can set up a hadoop cluster, do statistics on TBs worth of data, derive useful conclusions and speed it up by tweaking the low level data formats or microoptimizing the calculation. The issue is rarely paying these guys an extra $20k…

How hard can it be, though? Like taking a normal CS person and making them versatile with hadoop and so on? Could it be done for 20K$?

Making a CS person versatile with hadoop is not that hard. Making a CS person versatile with statistics is much harder.

See Zed Shaw's seminal article "Programmers Need To Learn Statistics Or I Will Kill Them All".

http://www.zedshaw.com/essays/programmer_stats.html

Making a math/science person versatile in CS is somewhat easier, but even that can be tricky. Many of them are bored by file formats, architecture, etc, and simply don't have the mindset of of engineering.

Re: Big Data's Big Problem: Little Talent

#62

Post by yummyfajitas » Sat, Apr 28, 2012, 4:13 PM UTC

groth wrote: post #48

yummyfajitas wrote: post #19

Earlier quoted context omitted.

Companies already do pay close to $200k/year for entry level data scientists. (what other kind of scientists are there? The tea-leaf reading kind?) "Data scientist" refers to the guy who can set up a hadoop cluster, do statistics on TBs worth of data, derive useful conclusions and speed it up by tweaking the low level data formats or microoptimizing the calculation. The issue is rarely paying these guys an extra $20k…

They do? Which companies? How do I find them? :p

Build a demo project showing you can do data analysis and they will find you.

Re: Big Data's Big Problem: Little Talent

#63

Post by ohyes » Sat, Apr 28, 2012, 4:14 PM UTC

_delirium wrote: post #3

I'm not sure that the kinds of employees that this article describes will ever be a large number. There could be more of them in the future, but someone who is top-notch at all of statistics, programming, and data-presentation has long been less common than someone who's good at one or two of those. Companies might consider looking at better ways to build teams that combine talent that exists, instead of pining for m…

I think another issue is that no one is interested in doing 'on the job' training. Few people learn statistics, programming, and data-presentation in college (not all three anyway). Companies might consider finding someone smart with one or two of the skills and expecting them to learn the other skills on the job. And what is talent? To me, talent is the ability to learn to do something quite well. To say there is a lack of 'talent,' when you have only searched for people who /already/ have those specific skills is disingenuous. You have barely looked.

Who is complaining about the lack of talent? Companies! Who produces talent? Companies! Who can fix this? Companies!

Part of running a company efficiently is is hiring, grooming and retaining juniors (who will become seniors). For God knows what reason, few companies actually understand this. If the skills you are looking for are lacking, hire someone smart and teach them.

Re: Big Data's Big Problem: Little Talent

#64

Post by achompas » Sat, Apr 28, 2012, 4:15 PM UTC

Tichy wrote: post #21

yummyfajitas wrote: post #19

Earlier quoted context omitted.

Companies already do pay close to $200k/year for entry level data scientists. (what other kind of scientists are there? The tea-leaf reading kind?) "Data scientist" refers to the guy who can set up a hadoop cluster, do statistics on TBs worth of data, derive useful conclusions and speed it up by tweaking the low level data formats or microoptimizing the calculation. The issue is rarely paying these guys an extra $20k…

How hard can it be, though? Like taking a normal CS person and making them versatile with hadoop and so on? Could it be done for 20K$?

How hard can it be?

Very hard. You run into all types of candidates who just aren't there yet: people working on research that's irrelevant to real world applications, people who have done data analysis/BI work that brand themselves as "data scientists," those who have the pedigree but cannot process and explore real-world data, those who have good analytical chops but not the distributed or advanced modeling experience, etc.

I've witnessed it first-hand, and it's tough to find the right person.

Re: Big Data's Big Problem: Little Talent

#65

Post by tosseraccount » Sat, Apr 28, 2012, 4:21 PM UTC

giardini wrote: post #41

Looks like the prelude to yet another H1-B buildup.

The only way we can really measure "shortage" is via compensation. If that's so, we need more hedge fund managers and surgeons, not grunt data crunchers. There are many problems with guest workers. The richest people in the world get special access to indentured labor. It targets specific industries thus amounting to a subsidy. It helps big business crush small business. The H1-B in particular is a tool to increase outsourcing and keep wages down (wages which are typically earned in the highest cost of living areas in the country at 60 hours a week). H1-B ? No, thank you! On the job training and good wages? Yes, please!

Re: Big Data's Big Problem: Little Talent

#66

Post by lgieron » Sat, Apr 28, 2012, 4:22 PM UTC

tomjen3 wrote: post #51

tgflynn wrote: post #31

Earlier quoted context omitted.

That's definitely not my world and one reason I left big-corp in the first place. EDIT ADDED: It seems like a really broken market if buyer decisions are completely orthogonal to the product being purchased.

I hope you are right. If you are, you have identified a potentially very, very lucrative option for a start-up. Broken markets can provide you a lot of money when you fix them.

This market is already (at least partially) covered by small consulting shops which provide sales front for competent freelancers who don't feel like doing the whole corporate networking&sales ritual.

Re: Big Data's Big Problem: Little Talent

#67

Post by ohashi » Sat, Apr 28, 2012, 4:23 PM UTC

ambiate wrote: post #54

I just listened to a lecture on this at BU. Emerging Internet Technologies at IBM or something of that nature. He was basically trying to sell us his product that crawled the internet (mainly a firehose at Twitter) and gathered statistics for advertisers and presented it in pretty graphics. The main issue they had was developing language recognition. Deciding if a user 'liked', 'loved', 'hated' or was 'neutral' about…

Maybe 200 is too small a sample size, but you can glean and predict stuff with that sort of data. My master's thesis was about predicting box office sales based off twitter data. (coincidentally these guys published a couple months before me: http://www.fastcompany.com/1604125/twitter-predicts-box-offi...) but we had incredibly similar results. It is pretty interesting to see what you can do with that data.

If it makes you feel any better, you can build it yourself, I certainly did. I didn't even use any libraries like NLTK to build my sentiment analysis. Read some research papers and built it from scratch (code wise at least, the ideas used were fairly common). It's a fun challenge. I still work with that code every day and use it in my startup now :)

Re: Big Data's Big Problem: Little Talent

#68

Post by _delirium » Sat, Apr 28, 2012, 4:24 PM UTC

ohyes wrote: post #63

_delirium wrote: post #3

I'm not sure that the kinds of employees that this article describes will ever be a large number. There could be more of them in the future, but someone who is top-notch at all of statistics, programming, and data-presentation has long been less common than someone who's good at one or two of those. Companies might consider looking at better ways to build teams that combine talent that exists, instead of pining for m…

I think another issue is that no one is interested in doing 'on the job' training. Few people learn statistics, programming, and data-presentation in college (not all three anyway). Companies might consider finding someone smart with one or two of the skills and expecting them to learn the other skills on the job. And what is talent? To me, talent is the ability to learn to do something quite well. To say there is a…

I agree somewhat, but retention is a fairly major problem. The shift away from career-length employment means that neither employers nor employees assume there will necessarily be much loyalty or longevity in the relationship. I think the decline in on-the-job training is directly related. Engineering firms used to be able to assume that it's okay to lose money on the first five years or so of an employee's work, if they built up skills that will make the company lots of money over the next 30-40 years of the employee's career. But if the company invests five years of significant training in a junior employee, and then the employee jumps ship to do freelance consulting or work for a competitor, the training never ends up paying for itself.

Re: Big Data's Big Problem: Little Talent

#69

Post by timr » Sat, Apr 28, 2012, 4:29 PM UTC

disgruntledphd2 wrote: post #5

_delirium wrote: post #3

I'm not sure that the kinds of employees that this article describes will ever be a large number. There could be more of them in the future, but someone who is top-notch at all of statistics, programming, and data-presentation has long been less common than someone who's good at one or two of those. Companies might consider looking at better ways to build teams that combine talent that exists, instead of pining for m…

I completely take most of your points, but I think that pretty much all quantitative PhD's are going to be close to "data scientists". Given that stats and explaining your research are requirements, all that's left is to train them to program, which a lot of people are already doing. As a matter of fact, since I heard about this big data stuff I've been honing my skills in this area, in case the hype actually manifes…

"I think that pretty much all quantitative PhD's are going to be close to "data scientists"."

Exactly. Fundamentally, "data science" is known far more widely by its other name: science. Yet we've reached the bizarro-world place where there are huge numbers of un- or under-employed scientists looking for work, while companies are freaking out about hiring the "rare" computer programmer who happens to know some statistics and self-identifies as a "data scientist". It's rather absurd.

Re: Big Data's Big Problem: Little Talent

#70

Post by gambler » Sat, Apr 28, 2012, 4:31 PM UTC

Managers frequently wail about skill shortages, but very often it's pure hypocrisy. The real problem is the reluctance to do any training (and I don't mean formal training) combined with the desire to get proven experts in whatever field. Proven experts must have years of experience in applying their expertise. If no one lets people with less experience to work in that field, where the hell would those experts appear from? Another dimension?

Can I be a 80% developer and 20% "data scientist" in your company to try the new role out? The bigger your company is, the less likely the answer to be a "yes". Since Big Data implies a big company, the resulting "shortage" is not surprising. It's self-made.