Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big Data's Big Problem: Little Talent

online.wsj.com

Post reply on HN
Print view View original

101–110 of 161 posts

Re: Big Data's Big Problem: Little Talent

#101

Post by mbucc »

kds wrote: post #37
mjw wrote: post #7

As an engineer who's investing in developing "deep expertise in statistics and machine learning" I can only stand to benefit from it, but something about the current wave of Big Data hype makes me instinctively a bit wary. Does this skills shortage really exist to the extent claimed? are there really enough people out there who would know what to do with a 'data scientist' if they were able to hire one? I see more ta…

I felt a similar kind of skepticism when I saw it took ~3 years to improve the Netflix recommendation system with just ~10% - in the context of the Netflix Prize, with great minds (data scientists and practitioners) participating and collaborating. Maybe the initial system was quite good and it had no space for easy-and-fast enhancements, I don't know. But 10% overall improvement result in 3 years (just as quantitati…

Last month Forbes reported that Netflix said 75% of what it's customers watch are from recommendations. Definitely some bang there.

Also, note that 10% improvement was far from linear: year 1: +8.43%, year2 +1% and year3 +0.6% (!!).

Re: Big Data's Big Problem: Little Talent

#102

Post by moonchrome »

pmb wrote: post #91

"claims of severe talent shortage in Big Data http://online.wsj.com/article/SB1000142405270230472330457736... Ok... where are the high salaries (500k$ a year)? No? No real shortage." https://twitter.com/#!/lemire/status/196245665951649793 Business has a shortage of "big data" folks in much the same way I have a "huge sailboat" shortage. Neither of us want to pay for it. We want it, but not for the going rate. Only on…

>We want it, but not for the going rate.

Does it cost 500k$/year for someone who got in that field to be in net positive ? I mean I know that people in US are complaining about high cost of higher education - but 500k$/year for it to be viable career path ? I think saying there is a shortage is justified, if the salaries are decent (and from anecdotal evidence I know that they are) people should be made aware that this field might be worth entering and that they should look in to it.

Re: Big Data's Big Problem: Little Talent

#103

Post by nahname »

mgkimsal wrote: post #20
_delirium wrote: post #3

I'm not sure that the kinds of employees that this article describes will ever be a large number. There could be more of them in the future, but someone who is top-notch at all of statistics, programming, and data-presentation has long been less common than someone who's good at one or two of those. Companies might consider looking at better ways to build teams that combine talent that exists, instead of pining for m…

I see this in software dev consulting, and it's probably in many other fields as well too. I see companies looking for one person who's a highly-skilled DBA, sysadmin, developer and who can interact affably with all levels of people in a company, including customers, at the drop of a hat. Often it's because they had one 'magical' person who did all that, though usually not very well, and the next person (or team) who…

There is a dark side to the separation of concerns. Lack of understanding. One person doing three jobs can fully understand what choices make the overal work more efficient. One DBA, one sysadmin and one dev only have direct insight into their own work. People, in my experience, can only optimize for what they understand and generally only care about their own work.

I believe this is why we are seeing roles, like devops becoming popular. Specialization introduces a communication overhead and most companies already do a terrible job of employee communication. Merging tightly coupled roles back together helps reduce friction and improve productivity.

In a small example such as your own, it probably works out closer to pipelining. The more common case, from my experience, is that the communication overhead and lack of understanding eat up more and more of your time as each new person is introduced. Law of diminishing returns takes effect and a year later everyone is always in meetings.

Re: Big Data's Big Problem: Little Talent

#104

Post by mitjak »

OzzyB wrote: post #55
bmelton wrote: post #43

Earlier quoted context omitted.

Perhaps slightly offtopic, but at least in Software dev consulting, there's a market for the multi-hatted individual in consulting directly, and pay is generally commensurate with the number of hats you can speak intelligently on to a customer. This isn't necessarily true everywhere -- when I lived in Memphis, TN, I often felt that I was dooming my professional career as, every time I ran into a challenge, I'd fork m…

I don't think your off-topic at all. I think your point speaks to the core of the issue at hand, which is, companies only want to hire folks who have mastered a particular skillset that they only deem important to them at that specific point-in-time . I don't for one minute think there aren't enough people who can't do "data driven analysis" or whatever it is that companies find important "today". Companies just don'…

The record industry analogy is brilliant. Major labels of the past were not unlike production lines. They weren't just management and promotion companies, but rather more like schools where promising resident talent was honed and music would be written for and played by most fitting artists. If one goes through hits of the 50-70s, they will find that the same songs by one label would often be performed by multiple artists, sometimes for years, before that final breakthrough performance would come along when the perfect match between the singer and the song would be found.

It was, then, a matter of knowing what to look for in new talent and finding ways to bring out their strengths through good and appropriate songwriting. People these days often complain about the quality of music being lower than it used to be, but the truth is that for every hit of the early times there would be many times the number of "duds" and just unsuccessful attempts that would either be mediocre songs or bad matchings between the artists and the records they perform.

IMO, despite even that, I still personally think records of the past are a lot more full of substance and soul than anything released today.

Re: Big Data's Big Problem: Little Talent

#105

Post by earl »

moonchrome wrote: post #102
pmb wrote: post #91

"claims of severe talent shortage in Big Data http://online.wsj.com/article/SB1000142405270230472330457736... Ok... where are the high salaries (500k$ a year)? No? No real shortage." https://twitter.com/#!/lemire/status/196245665951649793 Business has a shortage of "big data" folks in much the same way I have a "huge sailboat" shortage. Neither of us want to pay for it. We want it, but not for the going rate. Only on…

>We want it, but not for the going rate. Does it cost 500k$/year for someone who got in that field to be in net positive ? I mean I know that people in US are complaining about high cost of higher education - but 500k$/year for it to be viable career path ? I think saying there is a shortage is justified, if the salaries are decent (and from anecdotal evidence I know that they are) people should be made aware that th…

pmb is (correctly) saying that there is, by definition, no shortage of big data people. There's just a shortage at the (obviously below market clearing) price employers wish to pay. Also, you're ignoring the steep lead time to become a deep expert in stats / ML -- most likely a PhD plus significant programming time plus work experience.

Re: Big Data's Big Problem: Little Talent

#106

Post by earl »

NyxWulf wrote: post #16
mjw wrote: post #7

As an engineer who's investing in developing "deep expertise in statistics and machine learning" I can only stand to benefit from it, but something about the current wave of Big Data hype makes me instinctively a bit wary. Does this skills shortage really exist to the extent claimed? are there really enough people out there who would know what to do with a 'data scientist' if they were able to hire one? I see more ta…

As someone in the big data field on the ground (VP of Engineering). Let me give you my thoughts on it. Your impression about the hype is correct. There are a lot of vendors offering BIG solutions, if you pay them BIGGER money. Where I used to translate the word enterprise to $$, now I translate Big Data to $$$$$$$. When I'm hiring, I don't go looking for Big Data people, because generally they don't exist. Statistics…

@NyxWulf: where do you work / email me / curious

Re: Big Data's Big Problem: Little Talent

#107

Post by [deleted] »

robomartin wrote: post #93
_delirium wrote: post #3

I'm not sure that the kinds of employees that this article describes will ever be a large number. There could be more of them in the future, but someone who is top-notch at all of statistics, programming, and data-presentation has long been less common than someone who's good at one or two of those. Companies might consider looking at better ways to build teams that combine talent that exists, instead of pining for m…

I think you are very much on point with your comments. I am one of those people who are actually capable of going from MIG welding to designing websites, writing iOS apps, developing embedded hardware and software as well as GHz-range electronics with FPGA's, mechanical design and FEA. The only option for someone like me seems to be to run your own business. Nobody is likely to pay for the combined skill set. Which m…

[deleted]

Re: Big Data's Big Problem: Little Talent

#108

Post by kds »

tgflynn wrote: post #94
kds wrote: post #81

Earlier quoted context omitted.

10% increase in 3 years translates to around. 3.3% yearly growth rate. So in your example the 3.3% increase would be that 10M$ => so 1% of your annual business revenue is 10/3.3 or just above 3M$. But that means you already have a really significant business that makes ~300 M$ per year. And you manage to increase it just by peanuts (relatively speaking). And there is inflation in economy, and the alternative costs of…

Of course all these numbers are hypothetical, I have no idea what the actual Netflix numbers are, but aren't you assuming 100% profit margin ? If you actually have a machine learning application that increases annual revenue by 10% and your initial annual revenue is $300M (like in the example) and your profit margin is 50% then (neglecting the $1M cost of the model because it's small and amortized over many years) yo…

Well, the widely-known real example I mention is the Netflix Prize result -> 3.3% CAGR (cumulative annual growth rate, the 10% is for the whole period of 3 years, not for 1 year).

The numbers you mention are the hypothetical ones. Go and find a real publicly documented example that is close as values and margins to what you describe and I might agree.

NB. #1) Google and web-search, or some similar startup success story, as example doesn't count - they're the 2-person startup gone wildly successful, not a previously big entity that hired 2 ML-geniuses to open their eyes.

NB. #2) If I could bring a revolutionary increased value to a company - through vastly enhanced data processing and analysis - rather than doing a consultancy and educate somebody I'd rather enter the industry as competitor and prove the "old" guys don't understand the business anymore.

Re: Big Data's Big Problem: Little Talent

#109

Post by earl »

yummyfajitas wrote: post #19
tomjen3 wrote: post #2

Actually that is silly -- McKensey should now that there is and will never be a talent shortage. There will only be shortage of talent at a particular wage rate. If the companies paid newly graduated 'data-scientists' (what other kind of scientists are there? The tea-leaf reading kind?) 200k/year then they would have a lot more. It is pretty simple economics.

Companies already do pay close to $200k/year for entry level data scientists. (what other kind of scientists are there? The tea-leaf reading kind?) "Data scientist" refers to the guy who can set up a hadoop cluster, do statistics on TBs worth of data, derive useful conclusions and speed it up by tweaking the low level data formats or microoptimizing the calculation. The issue is rarely paying these guys an extra $20k…

Who is paying $200k for entry besides maybe google?

Re: Big Data's Big Problem: Little Talent

#110

Post by moonchrome »

earl wrote: post #105
moonchrome wrote: post #102

Earlier quoted context omitted.

>We want it, but not for the going rate. Does it cost 500k$/year for someone who got in that field to be in net positive ? I mean I know that people in US are complaining about high cost of higher education - but 500k$/year for it to be viable career path ? I think saying there is a shortage is justified, if the salaries are decent (and from anecdotal evidence I know that they are) people should be made aware that th…

pmb is (correctly) saying that there is, by definition, no shortage of big data people. There's just a shortage at the (obviously below market clearing) price employers wish to pay. Also, you're ignoring the steep lead time to become a deep expert in stats / ML -- most likely a PhD plus significant programming time plus work experience.

>by definition, no shortage of big data people

Shortage is defined by price being above the market equilibrium, we can debate what the equilibrium is but I think even at the current wages people should be interested in getting in to this field (I know a friend who is doing postgrad in math and interning for BI because the prospects are great), it's just that they can't get in fast enough - therefore IMO there is a temporary shortage.

>most likely a PhD plus significant programming time plus work experience.

Is it normal to expect 500k$/year for that experience in some other field ? I would sure like to know, maybe I can still switch :) I mean I know there are people making that kind of money but it can't be the average for PhD with work expirience ?

Post reply on HN