Big Data's Big Problem: Little Talent • Hacker News Board

Re: Big Data's Big Problem: Little Talent

#131

Post by bearmf » Sat, Apr 28, 2012, 9:19 PM UTC

jandrewrogers wrote: post #100

pmb wrote: post #91

"claims of severe talent shortage in Big Data http://online.wsj.com/article/SB1000142405270230472330457736... Ok... where are the high salaries (500k$ a year)? No? No real shortage." https://twitter.com/#!/lemire/status/196245665951649793 Business has a shortage of "big data" folks in much the same way I have a "huge sailboat" shortage. Neither of us want to pay for it. We want it, but not for the going rate. Only on…

The salaries are already moving north of $200k even outside of Silicon Valley and New York City and getting more expensive by the month. How high do they have to be before we have a "shortage"? The problem is not lack of money, it is that demand has greatly outstripped a finite supply. Very high wages do not automagically create new people with the requisite skills and this is the real bottleneck. It takes significan…

Look, this job title is at most 2 years old. How can someone have years of experience in this? OTOH, there are plenty of people with strong applied math and good programming skills.

Re: Big Data's Big Problem: Little Talent

#132

Post by seanharnett » Sat, Apr 28, 2012, 9:21 PM UTC

why-el wrote: post #75

So I was wondering if any fellow HNer is on a quest to be at least comfortable around these problems. Can you share your plans? Currently I am starting with some linear algebra and I have plans to move to statistics then pick up a book on machine learning. I would really use some advice.

Andrew Ng's machine learning class on coursera is a very nice, easy introduction to the subject.

Re: Big Data's Big Problem: Little Talent

#133

Post by why-el » Sat, Apr 28, 2012, 9:26 PM UTC

seanharnett wrote: post #132

why-el wrote: post #75

So I was wondering if any fellow HNer is on a quest to be at least comfortable around these problems. Can you share your plans? Currently I am starting with some linear algebra and I have plans to move to statistics then pick up a book on machine learning. I would really use some advice.

Andrew Ng's machine learning class on coursera is a very nice, easy introduction to the subject.

Yes, and I started that before realizing that I need more background knowledge, hence starting with some math. :)

Re: Big Data's Big Problem: Little Talent

#134

Post by achompas » Sat, Apr 28, 2012, 9:42 PM UTC

bearmf wrote: post #128

achompas wrote: post #64

Earlier quoted context omitted.

How hard can it be? Very hard. You run into all types of candidates who just aren't there yet: people working on research that's irrelevant to real world applications, people who have done data analysis/BI work that brand themselves as "data scientists," those who have the pedigree but cannot process and explore real-world data, those who have good analytical chops but not the distributed or advanced modeling experie…

If it is that hard the bar is probably set too high. Most of the skills are learned on the job after all. Most smart PhDs who can program well and have sound knowledge of statistics can learn to do this stuff.

Given enough time, anyone smart enough to finish a PhD can acquire a set of skills. :)

But it's more than just solid statistics. We're talking about having enough mathematical fluency to develop models rigorously (not just "oh, we'll minimize MSE!!"), test those models, then implement those models--possibly using a distributed algorithm.

From what I hear, these skills take years to develop. Choosing to groom the wrong person is an extremely costly mistake, so making the choice is difficult.

Re: Big Data's Big Problem: Little Talent

#135

Post by jandrewrogers » Sat, Apr 28, 2012, 10:23 PM UTC

bearmf wrote: post #131

jandrewrogers wrote: post #100

Earlier quoted context omitted.

The salaries are already moving north of $200k even outside of Silicon Valley and New York City and getting more expensive by the month. How high do they have to be before we have a "shortage"? The problem is not lack of money, it is that demand has greatly outstripped a finite supply. Very high wages do not automagically create new people with the requisite skills and this is the real bottleneck. It takes significan…

Look, this job title is at most 2 years old. How can someone have years of experience in this? OTOH, there are plenty of people with strong applied math and good programming skills.

The set of skills existed before it had a trendy job title so you can have the experience even if it was called something else. This is true of most of the people currently working as data scientists. In a similar vein, I was designing big data systems years before "big data" became a term or trendy. For any particular odd skill mix you can come up with, there are people with that skill mix who are already doing a similar job. But usually people do not intentionally build that skill mix until it becomes an official job title and career path in the eyes of the public so it is a very small pool of people.

In the case of modern data scientists, having strong applied mathematics and programming skills is about halfway to where you need to be and a good starting point. The demand has temporarily grown much faster than the convertible talent pool can develop the additional set of skills required.

Re: Big Data's Big Problem: Little Talent

#136

Post by reinhardt » Sat, Apr 28, 2012, 11:48 PM UTC

pgroves wrote: post #44

NyxWulf wrote: post #33

Earlier quoted context omitted.

I couldn't agree more. Accuracy is a problem, variation is another problem. Dealing with layers in the business who have no math or statistics background but very strong opinions is yet another complication. These types of conversations aren't uncommon. Other - "I need you to prove our stuff does X, Y, and Z". Me - "Ok.." Me - "Ok the data shows our stuff does X but Y and Z are just random noise" Other - "We ran it o…

Man, this is statistics. You should be able to get any result you want! I'm only half kidding. I can remember writing my first report (project summary) when I did a contract right after grad school. I put in maybe 5 graphs. Two looked good, three looked bad. The project manager just deleted the bad looking graphs and sent it on to the client.

"Some people use statistics like a drunk uses a lamp post - for support rather than for illumination."

Re: Big Data's Big Problem: Little Talent

#137

Post by thuzarsky » Sat, Apr 28, 2012, 11:55 PM UTC

ryanlchan wrote: post #77

disgruntledphd2 wrote: post #27

Earlier quoted context omitted.

Again, I completely see the difference (and am actually in the process of moving full time to the private sector from academia, so will probably understand a lot more in six months) but visualising data well is not that hard. Step 1: learn R Step 2: Learn PCA Step 3: Learn ggplot2 Step 4: play with the different geoms until you understand them (seriously though, everyone's eyes are optimised to find patterns, and if…

Allow a short vignette from a former academic and now management consultant. We spent six months at a major pharmaceuticals client examining their reimbursement data. Poring over many millions of rows of transaction data and thousands of payment codes (which, of course, were unique across sales geographies), we determined the ten regions at highest risk of reimbursement collapse. R was used, maps were created, beers…

Nailed it.

Re: Big Data's Big Problem: Little Talent

#138

Post by reinhardt » Sat, Apr 28, 2012, 11:56 PM UTC

earl wrote: post #123

anothermachine wrote: post #112

Earlier quoted context omitted.

Startups.* *Equity value, may vary unpredictably. And it's entry post-PhD, not entry from college.

So $120k and (very expensive) lottery tickets is what you're saying =P

$120K as a startup employee? Damn, I live in a wrong country.

Re: Big Data's Big Problem: Little Talent

#139

Post by bearmf » Sun, Apr 29, 2012, 12:10 AM UTC

achompas wrote: post #134

bearmf wrote: post #128

Earlier quoted context omitted.

If it is that hard the bar is probably set too high. Most of the skills are learned on the job after all. Most smart PhDs who can program well and have sound knowledge of statistics can learn to do this stuff.

Given enough time, anyone smart enough to finish a PhD can acquire a set of skills. :) But it's more than just solid statistics. We're talking about having enough mathematical fluency to develop models rigorously (not just "oh, we'll minimize MSE!!"), test those models, then implement those models--possibly using a distributed algorithm. From what I hear, these skills take years to develop. Choosing to groom the wron…

All mathematics consists of rigorous models. But choosing and tweaking a model is more of an art. Most data scientists apply existing models to new data, they do not develop new ones.

I am sure it takes much less than "years" for any smart PhD in applied mathematics to learn most of data analysis tricks. It is not theoretical physics after all.

Re: Big Data's Big Problem: Little Talent

#140

Post by achompas » Sun, Apr 29, 2012, 12:41 AM UTC

bearmf wrote: post #139

achompas wrote: post #134

Earlier quoted context omitted.

Given enough time, anyone smart enough to finish a PhD can acquire a set of skills. :) But it's more than just solid statistics. We're talking about having enough mathematical fluency to develop models rigorously (not just "oh, we'll minimize MSE!!"), test those models, then implement those models--possibly using a distributed algorithm. From what I hear, these skills take years to develop. Choosing to groom the wron…

All mathematics consists of rigorous models. But choosing and tweaking a model is more of an art. Most data scientists apply existing models to new data, they do not develop new ones. I am sure it takes much less than "years" for any smart PhD in applied mathematics to learn most of data analysis tricks. It is not theoretical physics after all.

Most data scientists apply existing models to new data, they do not develop new ones.

I meant "develop" in the software sense. Data scientists use off-the-shelf libraries during initial research, but those libraries usually lack an important feature preventing them from going into production (typically, no support for concurrency).

I am sure it takes much less than "years" ... to learn most of data analysis tricks.

I used to be cynical about "data science," too. After four months of working on a data science team, though, I'm a believer.

A data scientist is really a "full-stack data developer." He or she needs the ability to work with advanced models, use them to analyze large amounts of data, and modify those models to work concurrently or in a distributed system if desired (and its often desired). It's more than just "analysis tricks."