This is good news.
Big Data's Big Problem: Little Talent
141–150 of 161 posts
Re: Big Data's Big Problem: Little Talent
#142Earlier quoted context omitted.
I felt a similar kind of skepticism when I saw it took ~3 years to improve the Netflix recommendation system with just ~10% - in the context of the Netflix Prize, with great minds (data scientists and practitioners) participating and collaborating. Maybe the initial system was quite good and it had no space for easy-and-fast enhancements, I don't know. But 10% overall improvement result in 3 years (just as quantitati…
There's a fantastic paper, Hand 2006, which notes the strong tendency for simple models to get nearly all of the performance possible out of solvable problems. Hard problems do better with complex algorithms but there's also just less to be gained. The best solution tends to be simple models applied to the right kind of data such that the problem has become easy. This is sometimes pretty difficult though since the si…
Direct link : http://arxiv.org/pdf/math.ST/0606441.pdf
Re: Big Data's Big Problem: Little Talent
#143Earlier quoted context omitted.
Look, this job title is at most 2 years old. How can someone have years of experience in this? OTOH, there are plenty of people with strong applied math and good programming skills.
The set of skills existed before it had a trendy job title so you can have the experience even if it was called something else. This is true of most of the people currently working as data scientists. In a similar vein, I was designing big data systems years before "big data" became a term or trendy. For any particular odd skill mix you can come up with, there are people with that skill mix who are already doing a si…
I am of opinion that if demand is high enough, companies will start hiring "halfway there" people. But this will happen only if the market grows big enough. Right now it is still a niche market where companies are cherry-picking right candidates, it seems. At least this is the impression I get from reading this thread.
The question of the size of the market is crucial. Small labor markets are very inefficient. This means that the number of qualified people is small enough, but the number of companies they can choose from is also small. It is hard to find a job when the number of companies hiring is probably less than 100.
Re: Big Data's Big Problem: Little Talent
#144"claims of severe talent shortage in Big Data http://online.wsj.com/article/SB1000142405270230472330457736... Ok... where are the high salaries (500k$ a year)? No? No real shortage." https://twitter.com/#!/lemire/status/196245665951649793 Business has a shortage of "big data" folks in much the same way I have a "huge sailboat" shortage. Neither of us want to pay for it. We want it, but not for the going rate. Only on…
Even more, I don't notice an effort to expand the workforce by training or by recruitment of non-traditional workers, etc. The contra-logical statement "99% of programming applicants are unqualified" gets a lot of play in this field. But I would suggest something like "we can make 99% of applicants look like idiots with our circus-like hiring process". Yes, we've decided we have a shortage once we decide on five arbi…
I don't notice an effort to expand the workforce by training or by recruitment of non-traditional workers, etc.
"Code Year" for data scientists would actually need:
-- "Code Previous Year," where everyone would need to kick ass on Bayesian inference, linear algebra, and production-level software development; then,
-- during "Code Year," they'd proceed to learn about distributed algorithms, graphical models, and HMMs, then learn about distributed frameworks like Hadoop.
See Joseph Misti's comment here for more--it's really the most accurate list of skills one needs to become a data scientist:
http://www.quora.com/What-skills-are-needed-for-machine-lear...
Re: Big Data's Big Problem: Little Talent
#145Until the pay is comparable to finance, good luck?
I'd love to work on (arguably) cooler problems, but the combination of lower pay and the constant need to use the "hot new thing" to solve problems doesn't make transitioning look remotely attractive.
Really, the second is the HUGE obstacle: - You don't know anything about aNNs? Sorry, no job. - Nobody uses aNNs anymore, SVMs are all that matters. Sorry, come back after you catch up. - SVMs? Man, we need someone who's got expertise in optimizing RFs and Bayesian Trees. We don't want "black box" machine learning. We need to "understand" the results. Sorry no job. - Decision trees? GTFO man. We're doing rNNs now. - I'm pretty impressed with your data mining knowledge, but we're looking for someone with a background in DLMs and GPs. Sorry, no job. - repeat until vomit/suicide
I kind of wonder about the need for "badass" math skills; I'm not terribly convinced that math wizards are extraordinarily high value relative to people with other types of data analysis skills.
Re: Big Data's Big Problem: Little Talent
#146"claims of severe talent shortage in Big Data http://online.wsj.com/article/SB1000142405270230472330457736... Ok... where are the high salaries (500k$ a year)? No? No real shortage." https://twitter.com/#!/lemire/status/196245665951649793 Business has a shortage of "big data" folks in much the same way I have a "huge sailboat" shortage. Neither of us want to pay for it. We want it, but not for the going rate. Only on…
Even more, I don't notice an effort to expand the workforce by training or by recruitment of non-traditional workers, etc. The contra-logical statement "99% of programming applicants are unqualified" gets a lot of play in this field. But I would suggest something like "we can make 99% of applicants look like idiots with our circus-like hiring process". Yes, we've decided we have a shortage once we decide on five arbi…
That's a very good point: insisting that a single person must have skills in math, computer science and data interpretation is creating a purely arbitrary set of qualifications.
If we make a comparison with other fields, we can see that trying to find a single person who has skills in several diverse areas is not something that's usually done. For example, do companies try to hire bond traders who can implement their own trading software? Or do we insist that airline pilots or surgeons or CEOs should be able to build and repair the technology they use?
And if a company did manage to find a person who was both a good statistician and a good software developer, wouldn't the combination of responsibilities pull this person in too many directions, making it hard to focus on on what they were doing? Also, it would take a lot of effort to stay current with the latest developments in both math and computer science.
If a company was too small to be able to afford to hire three full-time specialists to analyze their data, they could outsource their data crunching needs to consulting companies that specialized in this kind of work.
I also have a problem with the newly-coined term "data science". Scientists are engaged in discovering fundamental truths about the way the physical world works. I don't think that finding trends in a company's data counts as science. (I don't think that 99% of "computer scientists" are scientists either, including the professors I knew in grad school.) I liked the older term "data mining" much better, but I guess it's not trendy enough anymore.
Re: Big Data's Big Problem: Little Talent
#147I'm not sure that the kinds of employees that this article describes will ever be a large number. There could be more of them in the future, but someone who is top-notch at all of statistics, programming, and data-presentation has long been less common than someone who's good at one or two of those. Companies might consider looking at better ways to build teams that combine talent that exists, instead of pining for m…
Good take on the article. I.e., there's a sense that money is to be made through "big data" if only we could find the right, highly skilled person.
Now, we notice that we're not making those gobs of money, as promised in the business press (and shown through a few example successes), and conclude that there must be a shortage of such people.
Re: Big Data's Big Problem: Little Talent
#148As an engineer who's investing in developing "deep expertise in statistics and machine learning" I can only stand to benefit from it, but something about the current wave of Big Data hype makes me instinctively a bit wary. Does this skills shortage really exist to the extent claimed? are there really enough people out there who would know what to do with a 'data scientist' if they were able to hire one? I see more ta…
As someone in the big data field on the ground (VP of Engineering). Let me give you my thoughts on it. Your impression about the hype is correct. There are a lot of vendors offering BIG solutions, if you pay them BIGGER money. Where I used to translate the word enterprise to $$, now I translate Big Data to $$$$$$$. When I'm hiring, I don't go looking for Big Data people, because generally they don't exist. Statistics…
Some years ago, there was a similar wave of enthusiasm for "data mining." Plus ça change...
Re: Big Data's Big Problem: Little Talent
#149"claims of severe talent shortage in Big Data http://online.wsj.com/article/SB1000142405270230472330457736... Ok... where are the high salaries (500k$ a year)? No? No real shortage." https://twitter.com/#!/lemire/status/196245665951649793 Business has a shortage of "big data" folks in much the same way I have a "huge sailboat" shortage. Neither of us want to pay for it. We want it, but not for the going rate. Only on…
The salaries are already moving north of $200k even outside of Silicon Valley and New York City and getting more expensive by the month. How high do they have to be before we have a "shortage"? The problem is not lack of money, it is that demand has greatly outstripped a finite supply. Very high wages do not automagically create new people with the requisite skills and this is the real bottleneck. It takes significan…
Finally, some basic labor market economics. Just like how employers have restrictions on the number of people they are able to hire, employees have restrictions on the number of hours they are able to work. Or for the sake of this example, whether they are able to be a data scientist or not. The aspiring data scientists' restrictions all have to do with their ability. And as you point out, the cost of human capital investment in this area is very, very high.
One doesn't need to be an econometric theorist to be a data scientist, but I feel that far too few hackers truly appreciate the elegance of some standard, say, 1st year economics grad school econometric models. Things like panel models, IVs, 2SLS, GMM and even just taking seriously the basic assumptions of OLS regression -- there's a reason most econometrics classes (grad or undergrad) always start with the ~5 assumptions of OLS regression.
TL;DR Economists would make great data scientists (and better economists) if only they understood and appreciated computer science more.
Re: Big Data's Big Problem: Little Talent
#150Earlier quoted context omitted.
This technology will change the world more than the Internet or any other technology in human history. You're right that adoption is very slow. I'm convinced that businesses could save trillions of dollars by applying existing weak AI to their problems. Why aren't they doing it ? For one thing there's a huge gulf between the average business person's understanding of what is possible and what actually is. On the othe…
I have had a website and a LinkedIn profile for over a year that make this fairly clear It doesn't work like that. You LinkedIn profile might easily land you any job in Software development, but not consulting. In my opinion, if you want to do consulting for big corp. you should figure out what it takes to it. An attractive website and presentation, few buzzwords, client testimonials, business cards, and the other bl…
Big Corp - "We can't find people who can do X!!!!"
Person who can do X - "I've been saying over here I can do X."
Big Corp - "Oh. We don't look over there, it's not how it's done."
I think I'm seeing part of the issue here.