Big Data's Big Problem: Little Talent • Hacker News Board

Re: Big Data's Big Problem: Little Talent

#111

Post by anothermachine » Sat, Apr 28, 2012, 7:15 PM UTC

tomjen3 wrote: post #59

enfilade wrote: post #45

Earlier quoted context omitted.

I agree with your overall line of thinking. In your last sentence you wrote: "There is a shortage of talent in computer science, but never in the other disciplines, it may take another 30 years for the suits to have the ability to understand why." Could you elaborate on this point -- do you feel that there is not a shortage in disciplines such as dentistry and law because many people are willing to work very hard for…

You can actually make good money by going into the law or medicine. You have to work and and be skilled of course, but lets be honest that is also required for a start-up. I can't help but think that ability is because you can't run a law firm without being a lawyer so the boss has some idea of what it means to be a good lawyer and how to treat them.

Most lawyers at the top-income end hate their bosses and jobs. Law firm partnership track at large firms is a dog-eat-dog 80-hour week hell. People are only happy when they are the few who claw to the top, or the many who drop out. The ones in the middle are suffering as bad as any stereotypical bank programmer.

Re: Big Data's Big Problem: Little Talent

#112

Post by anothermachine » Sat, Apr 28, 2012, 7:17 PM UTC

earl wrote: post #109

yummyfajitas wrote: post #19

Earlier quoted context omitted.

Companies already do pay close to $200k/year for entry level data scientists. (what other kind of scientists are there? The tea-leaf reading kind?) "Data scientist" refers to the guy who can set up a hadoop cluster, do statistics on TBs worth of data, derive useful conclusions and speed it up by tweaking the low level data formats or microoptimizing the calculation. The issue is rarely paying these guys an extra $20k…

Who is paying $200k for entry besides maybe google?

Startups.*

*Equity value, may vary unpredictably.

And it's entry post-PhD, not entry from college.

Re: Big Data's Big Problem: Little Talent

#113

Post by anothermachine » Sat, Apr 28, 2012, 7:19 PM UTC

tomjen3 wrote: post #50

Tichy wrote: post #6

Earlier quoted context omitted.

Isn't that point of view also colored by ideology? Even supposing there are enough people capable of becoming that kind of "talent", what if those talents are also sought for in other kinds of jobs? Granted, if it were really urgent, perhaps companies would start looking in the most remote places for talents, so with a population of 6 billion perhaps there really would be enough who could be trained. How many of thos…

If those talents are also sought after for other jobs then the price will go up until one of the jobs will be done by some other method or some other person. I do have trouble imagining that anybody who is working on a farm would be a good data-scientist but then I no very, very little about farming. Economics is not tainted or colored by ideology, it is a science. It is the study of how best to allocate limited reso…

> Economics is not tainted or colored by ideology,

You must be joking. Economics is the most ideological of the sciences, because so much of it cannot be tested in nature or a lab, as it only can be tested an impractically large scale and with many confounding factors.

Re: Big Data's Big Problem: Little Talent

#114

Post by kds » Sat, Apr 28, 2012, 7:23 PM UTC

mbucc wrote: post #101

kds wrote: post #37

Earlier quoted context omitted.

I felt a similar kind of skepticism when I saw it took ~3 years to improve the Netflix recommendation system with just ~10% - in the context of the Netflix Prize, with great minds (data scientists and practitioners) participating and collaborating. Maybe the initial system was quite good and it had no space for easy-and-fast enhancements, I don't know. But 10% overall improvement result in 3 years (just as quantitati…

Last month Forbes reported that Netflix said 75% of what it's customers watch are from recommendations. Definitely some bang there. Also, note that 10% improvement was far from linear: year 1: +8.43%, year2 +1% and year3 +0.6% (!!).

An interesting observation, indeed - so the enhancement opportunities were effectively explored within the first year or so. The next two years count for far less, though I guess the cleverest approaches started to emerge just then.

Re: Big Data's Big Problem: Little Talent

#115

Post by mgkimsal » Sat, Apr 28, 2012, 7:24 PM UTC

nahname wrote: post #103

mgkimsal wrote: post #20

Earlier quoted context omitted.

I see this in software dev consulting, and it's probably in many other fields as well too. I see companies looking for one person who's a highly-skilled DBA, sysadmin, developer and who can interact affably with all levels of people in a company, including customers, at the drop of a hat. Often it's because they had one 'magical' person who did all that, though usually not very well, and the next person (or team) who…

There is a dark side to the separation of concerns. Lack of understanding. One person doing three jobs can fully understand what choices make the overal work more efficient. One DBA, one sysadmin and one dev only have direct insight into their own work. People, in my experience, can only optimize for what they understand and generally only care about their own work. I believe this is why we are seeing roles, like dev…

There's a balance to be struck. Specialization brings with is tunnel-vision, and having the bulk of people on a dev team have a decent understanding of the other parts of the stack is certainly useful to help avoiding that.

Jack-of-all-trades devs have a place, but dedicated people in specific roles also have a place. Those places will change and move over time as the nature of the project and the business changes (initial dev in to maintenance, early market upstart vs established leader, etc). Understanding and accepting that role changes may be necessary is probably the hardest thing for some business to accept, and properly making those changes (filling roles with good hires) is arguably one of the hardest things to execute on.

Re: Big Data's Big Problem: Little Talent

#116

Post by pfedor » Sat, Apr 28, 2012, 7:30 PM UTC

earl wrote: post #105

moonchrome wrote: post #102

Earlier quoted context omitted.

>We want it, but not for the going rate. Does it cost 500k$/year for someone who got in that field to be in net positive ? I mean I know that people in US are complaining about high cost of higher education - but 500k$/year for it to be viable career path ? I think saying there is a shortage is justified, if the salaries are decent (and from anecdotal evidence I know that they are) people should be made aware that th…

pmb is (correctly) saying that there is, by definition, no shortage of big data people. There's just a shortage at the (obviously below market clearing) price employers wish to pay. Also, you're ignoring the steep lead time to become a deep expert in stats / ML -- most likely a PhD plus significant programming time plus work experience.

Of course, by that definition, there is never a shortage of anything.

Re: Big Data's Big Problem: Little Talent

#117

Post by ced » Sat, Apr 28, 2012, 7:37 PM UTC

jandrewrogers wrote: post #98

tomjen3 wrote: post #2

Actually that is silly -- McKensey should now that there is and will never be a talent shortage. There will only be shortage of talent at a particular wage rate. If the companies paid newly graduated 'data-scientists' (what other kind of scientists are there? The tea-leaf reading kind?) 200k/year then they would have a lot more. It is pretty simple economics.

Bags of money are already being waved around, that is not the problem. Wages are already moving north of $200k for these positions because you can't find people with the basic skills for any amount of money. Being a "data scientist" as currently defined in practice requires someone to be a polymath with skills that are individually high value and not commonly found together. Roughly speaking, you need some aptitude a…

Computational geometry??? That's a new one for me. Do you mean only linear/convex programming?

Incidentally, I would really like to hear about the kind of Real Work that data scientists end up doing with TBs of data, because I'm always fuzzy on the details. MCMC? Variational methods? SVMs? Or is it more oriented towards frequentist statistical methods, applied at "web-scale"?

Re: Big Data's Big Problem: Little Talent

#118

Post by ced » Sat, Apr 28, 2012, 7:40 PM UTC

lgieron wrote: post #66

tomjen3 wrote: post #51

Earlier quoted context omitted.

I hope you are right. If you are, you have identified a potentially very, very lucrative option for a start-up. Broken markets can provide you a lot of money when you fix them.

This market is already (at least partially) covered by small consulting shops which provide sales front for competent freelancers who don't feel like doing the whole corporate networking&sales ritual.

Where do I find these small consulting shops?

Re: Big Data's Big Problem: Little Talent

#119

Post by arjunnarayan » Sat, Apr 28, 2012, 7:47 PM UTC

pfedor wrote: post #116

earl wrote: post #105

Earlier quoted context omitted.

pmb is (correctly) saying that there is, by definition, no shortage of big data people. There's just a shortage at the (obviously below market clearing) price employers wish to pay. Also, you're ignoring the steep lead time to become a deep expert in stats / ML -- most likely a PhD plus significant programming time plus work experience.

Of course, by that definition, there is never a shortage of anything.

This is a very good quersion, and made me think a little. Here's my stab at it:

If we define the shortage as "shortage of people willing to do X for $200,000 a year", that's clearly a bad definition. You should just pay more (as earl suggested) to get what you want. But what if that's just not possible on a macro level?

Consider if you have an aggregate demand of "the market needs a total of 500 Data scientists". If there are only 250 data scientists in the world, their salaries will be bid up, then I can see somebody crying that there's a shortage for affordable data scientists (whatever that means). But any capitalist will tell you that they're just looking for a free sailboat. On the other hand, the 250 data scientists are being paid a lot, and on some level skills are somewhat fungible, so you end up with non-data scientists (maybe vanilla statisticians/actuaries) moving sideways to get in on this payday. So you have some retraining, and in the long run things tend to work out. So there's no shortage.

But in the long run we are all dead. Thus even if we define a shortage as the shortfall in supply at _any_ price, this isn't sufficient. In the short run there can very well be a shortfall.

If it takes 3 years to train a data scientist (I'm just making stuff up here), and there are 250 data scientists on the market, if you have an aggregate demand for 500 data scientists _today_ --- completely price unconditional --- you just cannot fill it. Price is almost irrelevant (on the macro level --- you as an individual can always outbid your competitors). There is a temporal shortage that cannot be filled.

I think this is the precise definition of what a shortage is that you are looking for. Shortages exist in the macro scale. Shortages do not exist for individual companies (they should just pay more, and if the benefits are not worth the cost of hiring, there isn't a shortage, they're just cheap).

Re: Big Data's Big Problem: Little Talent

#120

Post by achompas » Sat, Apr 28, 2012, 7:49 PM UTC

why-el wrote: post #75

So I was wondering if any fellow HNer is on a quest to be at least comfortable around these problems. Can you share your plans? Currently I am starting with some linear algebra and I have plans to move to statistics then pick up a book on machine learning. I would really use some advice.

I'm on this path right now. Working simultaneously on a MSCS at NYU and a full-time developer gig at Knewton (a company which truly understands the value and risks of data R&D).

I'd really start with this awesome curriculum[0] by Joseph Misti. He nails the mix of modeling, algorithms, math/stats, development, and distributed/Unix skills one needs to become comfortable around these problems. More importantly, his advice agrees with my experience on the data team at Knewton--we really use a bit of all of the above skills to solve our problems.

My email is in my profile if you (or anyone else) would like to chat about this a bit more. I'm also in NYC, and totally willing to grab tea and chat in person.

[0] http://www.quora.com/What-skills-are-needed-for-machine-lear...