Big Data's Big Problem: Little Talent • Hacker News Board

Re: Big Data's Big Problem: Little Talent

#81

Post by kds » Sat, Apr 28, 2012, 4:57 PM UTC

tgflynn wrote: post #39

kds wrote: post #37

Earlier quoted context omitted.

I felt a similar kind of skepticism when I saw it took ~3 years to improve the Netflix recommendation system with just ~10% - in the context of the Netflix Prize, with great minds (data scientists and practitioners) participating and collaborating. Maybe the initial system was quite good and it had no space for easy-and-fast enhancements, I don't know. But 10% overall improvement result in 3 years (just as quantitati…

But what if 10% improvement means 10 M$/year ? Anyway I think there are many applications where getting the absolute best performance isn't as important as finding the problem, figuring out how to apply a machine learning model to it (which includes getting the necessary training data) then training an off the shelf mode. The later of these may take a day or less, the other phases may well require both more thought a…

10% increase in 3 years translates to around. 3.3% yearly growth rate. So in your example the 3.3% increase would be that 10M$ => so 1% of your annual business revenue is 10/3.3 or just above 3M$.

But that means you already have a really significant business that makes ~300 M$ per year. And you manage to increase it just by peanuts (relatively speaking).

And there is inflation in economy, and the alternative costs of not investing such a huge sum or part of in Apple stocks (for example) during those years.

My point explained better:

The startup success of getting from zero to millions just because of clever ML/data-science/statistics is something to be respected and admired. But for already big-business all this big-data buzz might provide just minor enhancement opportunities at best.

Re: Big Data's Big Problem: Little Talent

#82

Post by chimeracoder » Sat, Apr 28, 2012, 5:02 PM UTC

disgruntledphd2 wrote: post #5

_delirium wrote: post #3

I'm not sure that the kinds of employees that this article describes will ever be a large number. There could be more of them in the future, but someone who is top-notch at all of statistics, programming, and data-presentation has long been less common than someone who's good at one or two of those. Companies might consider looking at better ways to build teams that combine talent that exists, instead of pining for m…

I completely take most of your points, but I think that pretty much all quantitative PhD's are going to be close to "data scientists". Given that stats and explaining your research are requirements, all that's left is to train them to program, which a lot of people are already doing. As a matter of fact, since I heard about this big data stuff I've been honing my skills in this area, in case the hype actually manifes…

> I think that pretty much all quantitative PhD's are going to be close to "data scientists"

Having taken all but one of the core requirements for a masters' degree in statistics at a university with a well-respected statistics department, I can tell you that's very much not true.

The true challenges in data science have almost nothing to do with what you spend 90% of your time as a graduate student studying (whether you're getting an MA or a PhD, this applies the same). You may happen to end up a qualified data scientist, but that's not by design of the program.

The big problems in data science are almost a disjoint set from the big problems in statistics (at least the solved ones), and that's because the things that are tractable from a theoretical/mathematical perspective are very different from the ones that we hope to solve in the workforce. We're just starting to bridge this gap in recent years (particularly with the advent of computers), but that's a very, very nascent trend.

This isn't unique to my university, either - most schools just simply aren't teaching the type of skills that a data scientist - not a statistician, but a data scientist - would need to be competitive in the work force. Those that do know these skills mostly do by chance - either because they branched into statistics from another discipline, because they were forced to learn it on the job, or because they took the time to learn it themselves.

All three of those are pretty rare - I recently took a class in applied data mining and Bayesian statistics. Except for a few undergraduates majoring in comp sci, the class was mostly graduate students in statistics, and those who knew how to program were in the stark minority (and were very popular when we were picking project groups!)

> all that's left is to train them to program

And to turn everything that they've learned and studied for the past two, four, or more years on its head so that they can actually put it to use. Okay, not everything, but at least 80% of it. Seriously, studying statistics at a high level is incredibly valuable, but it's not sufficient - it's not even going to get you half of the way there.

Re: Big Data's Big Problem: Little Talent

#83

Post by mr-ron » Sat, Apr 28, 2012, 5:03 PM UTC

_delirium wrote: post #68

ohyes wrote: post #63

Earlier quoted context omitted.

I think another issue is that no one is interested in doing 'on the job' training. Few people learn statistics, programming, and data-presentation in college (not all three anyway). Companies might consider finding someone smart with one or two of the skills and expecting them to learn the other skills on the job. And what is talent? To me, talent is the ability to learn to do something quite well. To say there is a…

I agree somewhat, but retention is a fairly major problem. The shift away from career-length employment means that neither employers nor employees assume there will necessarily be much loyalty or longevity in the relationship. I think the decline in on-the-job training is directly related. Engineering firms used to be able to assume that it's okay to lose money on the first five years or so of an employee's work, if…

What prevented employees from jumping ship before? Was there better long term benefits associated with staying with a company?

Re: Big Data's Big Problem: Little Talent

#84

Post by ohyes » Sat, Apr 28, 2012, 5:05 PM UTC

_delirium wrote: post #68

ohyes wrote: post #63

Earlier quoted context omitted.

I think another issue is that no one is interested in doing 'on the job' training. Few people learn statistics, programming, and data-presentation in college (not all three anyway). Companies might consider finding someone smart with one or two of the skills and expecting them to learn the other skills on the job. And what is talent? To me, talent is the ability to learn to do something quite well. To say there is a…

I agree somewhat, but retention is a fairly major problem. The shift away from career-length employment means that neither employers nor employees assume there will necessarily be much loyalty or longevity in the relationship. I think the decline in on-the-job training is directly related. Engineering firms used to be able to assume that it's okay to lose money on the first five years or so of an employee's work, if…

That's definitely true, but I think that a part of on the job training is building loyalty. If you like the people you work with and the salary and benefits are pretty decent, you are not likely to want to go looking for another job.

For my last job (at a really big company), the only time that substantially increasing my salary came up was when I was already on my way out the door, and they realized 'oh shit, we really depend on this guy' and tried to counter offer.

I would get glowing reviews but the maximum my salary could possibly ever increase in a year was 5%. Changing companies it could increase by as much as 50-60%. So we can say that it is a 'bad relationship' but mostly it is simple math. Of course you are going to have to pay the highly skilled person a salary that is commensurate to their skills. Additionally, you should work to keep them at the salary the market will bear rather than waiting for them to get a better offer from someone else. It will cut into your profit margin, but it is a lot better than being on the defensive and having to counter offer against a company that the employee has already talked herself into wanting to be at.

Re: Big Data's Big Problem: Little Talent

#85

Post by _delirium » Sat, Apr 28, 2012, 5:16 PM UTC

mr-ron wrote: post #83

_delirium wrote: post #68

Earlier quoted context omitted.

I agree somewhat, but retention is a fairly major problem. The shift away from career-length employment means that neither employers nor employees assume there will necessarily be much loyalty or longevity in the relationship. I think the decline in on-the-job training is directly related. Engineering firms used to be able to assume that it's okay to lose money on the first five years or so of an employee's work, if…

What prevented employees from jumping ship before? Was there better long term benefits associated with staying with a company?

Yes, on the latter point: vacation time and pensions typically increased based on length of service with the company. You ended up with a much worse pension if you had 10 years' service with each of four companies, than if you had 40 years' service with one company, under the traditional defined-benefit pension schemes.

There are probably a lot of cultural changes influencing it though, perhaps more; changing jobs frequently as a salaried professional just wasn't something many people in my dad's generation actively considered. One of many factors might be the change in how promotions are done; it used to much more often be "within the ranks". You worked your way up to FooBar VP or even FooBar CEO by starting in a regular job and getting promoted up the ladder, which required staying at the company for a long time. Now it's more common to hire external people right into senior posts.

Re: Big Data's Big Problem: Little Talent

#86

Post by alecco » Sat, Apr 28, 2012, 5:17 PM UTC

giardini wrote: post #41

Looks like the prelude to yet another H1-B buildup.

Only this time US is less attractive. Even mexicans started to move out.

Re: Big Data's Big Problem: Little Talent

#87

Post by tel » Sat, Apr 28, 2012, 5:30 PM UTC

chimeracoder wrote: post #82

disgruntledphd2 wrote: post #5

Earlier quoted context omitted.

I completely take most of your points, but I think that pretty much all quantitative PhD's are going to be close to "data scientists". Given that stats and explaining your research are requirements, all that's left is to train them to program, which a lot of people are already doing. As a matter of fact, since I heard about this big data stuff I've been honing my skills in this area, in case the hype actually manifes…

> I think that pretty much all quantitative PhD's are going to be close to "data scientists" Having taken all but one of the core requirements for a masters' degree in statistics at a university with a well-respected statistics department, I can tell you that's very much not true. The true challenges in data science have almost nothing to do with what you spend 90% of your time as a graduate student studying (whether…

I'm in the same boat and one of the funnier professors in math stat loves to talk about the students he hasn't "ruined" because they manage to learn programming and practical finite sample wisdom and go on to be successful in the industry.

And then he talks about his other students, with great love, who just like proving theorems.

Re: Big Data's Big Problem: Little Talent

#88

Post by tel » Sat, Apr 28, 2012, 5:43 PM UTC

kds wrote: post #37

mjw wrote: post #7

As an engineer who's investing in developing "deep expertise in statistics and machine learning" I can only stand to benefit from it, but something about the current wave of Big Data hype makes me instinctively a bit wary. Does this skills shortage really exist to the extent claimed? are there really enough people out there who would know what to do with a 'data scientist' if they were able to hire one? I see more ta…

I felt a similar kind of skepticism when I saw it took ~3 years to improve the Netflix recommendation system with just ~10% - in the context of the Netflix Prize, with great minds (data scientists and practitioners) participating and collaborating. Maybe the initial system was quite good and it had no space for easy-and-fast enhancements, I don't know. But 10% overall improvement result in 3 years (just as quantitati…

There's a fantastic paper, Hand 2006, which notes the strong tendency for simple models to get nearly all of the performance possible out of solvable problems.

Hard problems do better with complex algorithms but there's also just less to be gained.

The best solution tends to be simple models applied to the right kind of data such that the problem has become easy. This is sometimes pretty difficult though since the simple models are designed on simple data, which might not always be what you've got.

Re: Big Data's Big Problem: Little Talent

#89

Post by read_wharf » Sat, Apr 28, 2012, 5:44 PM UTC

gaius wrote: post #79

randomStuff wrote: post #42

Earlier quoted context omitted.

This is a total ploy by large companies to increase the H1-B visa cap. I have seen companies post job openings with starting salaries of 40K for experienced developer positions so they can then claim there were no American applicants. Its a self fulfilling prophecy, if companies outsource the jobs then people don't study those skills for fear of their job going to India; then the companies complain that there aren't…

One other thing is that the traditional entry route to software for the non-traditional candidate was via the helpdesk or QA department. You got your foot in the door, impressed the established engineers by turning around tickets quickly or by writing comprehensive bug reports, then when they needed another developer, you got the tap on the shoulder. Some of the best engineers I've worked with have come in through th…

If dev is outsourced, then they'll be right there in the building when they need to tap a QA shoulder.

Re: Big Data's Big Problem: Little Talent

#90

Post by JeffBlauser » Sat, Apr 28, 2012, 5:46 PM UTC

bmelton wrote: post #43

mgkimsal wrote: post #20

Earlier quoted context omitted.

I see this in software dev consulting, and it's probably in many other fields as well too. I see companies looking for one person who's a highly-skilled DBA, sysadmin, developer and who can interact affably with all levels of people in a company, including customers, at the drop of a hat. Often it's because they had one 'magical' person who did all that, though usually not very well, and the next person (or team) who…

Perhaps slightly offtopic, but at least in Software dev consulting, there's a market for the multi-hatted individual in consulting directly, and pay is generally commensurate with the number of hats you can speak intelligently on to a customer. This isn't necessarily true everywhere -- when I lived in Memphis, TN, I often felt that I was dooming my professional career as, every time I ran into a challenge, I'd fork m…

That's good to hear you found your niche. I sometimes look back with pangs of regret, to be honest. When my compan(ies) needed visual basic help, I figured it out. Then never used it again. Then repeated that over and over with asp, php, sas, sysadmin, graphic design, crm, erp, even non-tech things such as accounting, loss prevention in varying degrees... So I can't really apply as an 'expert' at any. I don't dwell on it, but I am not sure if I would do it exactly the same way again.