Big Data's Big Problem: Little Talent
online.wsj.com
Big Data's Big Problem: Little Talent
1–10 of 161 posts
Re: Big Data's Big Problem: Little Talent
#2Actually that is silly -- McKensey should now that there is and will never be a talent shortage. There will only be shortage of talent at a particular wage rate.
If the companies paid newly graduated 'data-scientists' (what other kind of scientists are there? The tea-leaf reading kind?) 200k/year then they would have a lot more. It is pretty simple economics.
Re: Big Data's Big Problem: Little Talent
#3I'm not sure that the kinds of employees that this article describes will ever be a large number. There could be more of them in the future, but someone who is top-notch at all of statistics, programming, and data-presentation has long been less common than someone who's good at one or two of those. Companies might consider looking at better ways to build teams that combine talent that exists, instead of pining for more superstars.
I'm reminded indirectly of an acquaintance of mine who works on repairing industrial machinery, where companies complain of a big skills shortage. They either fail to realize or are in denial about what that means in the 21st century, though. It might've been a one-person job in the 1950s, a skilled-labor type of repairman job. But today they want to find one person who can do the physical work (welding, etc.), EE type work, embedded-systems programming (and possibly reverse engineering), application-level programming to hook things up to their network, etc. Some of these people exist, but it's more common to find boutique consulting firms with 3-person teams of EE/CE/machinist or some such permutation. But companies balk at paying consulting fees equivalent to three professional salaries for something they think "should" be doable by one person with a magical combination of skills, who will work for maybe $80k. So they complain that there is a shortage of people who can repair truck scales (for example).
Re: Big Data's Big Problem: Little Talent
#4Actually that is silly -- McKensey should now that there is and will never be a talent shortage. There will only be shortage of talent at a particular wage rate. If the companies paid newly graduated 'data-scientists' (what other kind of scientists are there? The tea-leaf reading kind?) 200k/year then they would have a lot more. It is pretty simple economics.
You're absolutely right. That said, I don't buy that big-data is as revolutionary as the Internet. While in theory, every single business can collect data and optimize based on what they see, this is way too complex for most businesses to deal with. While big data has certainly been critical for the business model of ad-based startups, I don't see it being used in other industries. People keep alluding to data-driven medicine and genetic analyses. These are some of the most complex information analyses domains, and yet, I don't see benefit commensurate with the big data hype. I'd love to hear counter examples though!
Re: Big Data's Big Problem: Little Talent
#5I'm not sure that the kinds of employees that this article describes will ever be a large number. There could be more of them in the future, but someone who is top-notch at all of statistics, programming, and data-presentation has long been less common than someone who's good at one or two of those. Companies might consider looking at better ways to build teams that combine talent that exists, instead of pining for m…
I completely take most of your points, but I think that pretty much all quantitative PhD's are going to be close to "data scientists". Given that stats and explaining your research are requirements, all that's left is to train them to program, which a lot of people are already doing. As a matter of fact, since I heard about this big data stuff I've been honing my skills in this area, in case the hype actually manifests.
Re: Big Data's Big Problem: Little Talent
#6Actually that is silly -- McKensey should now that there is and will never be a talent shortage. There will only be shortage of talent at a particular wage rate. If the companies paid newly graduated 'data-scientists' (what other kind of scientists are there? The tea-leaf reading kind?) 200k/year then they would have a lot more. It is pretty simple economics.
Isn't that point of view also colored by ideology? Even supposing there are enough people capable of becoming that kind of "talent", what if those talents are also sought for in other kinds of jobs?
Granted, if it were really urgent, perhaps companies would start looking in the most remote places for talents, so with a population of 6 billion perhaps there really would be enough who could be trained. How many of those 6 billions are "free" in a sense, as in not needed for maintenance of human life (farming, medicine, building shelter and so on)?
But do economics really work that way? Could we extrapolate that logic to conclude that there is no problem in the world at all? All it takes is enough money to solve every problem - alas, the money doesn't seem to be there, or allocating it properly is apparently hard. (Hm, some of those talents might be able to help, for a true bootstrap solution).
Re: Big Data's Big Problem: Little Talent
#7As an engineer who's investing in developing "deep expertise in statistics and machine learning" I can only stand to benefit from it, but something about the current wave of Big Data hype makes me instinctively a bit wary.
Does this skills shortage really exist to the extent claimed? are there really enough people out there who would know what to do with a 'data scientist' if they were able to hire one? I see more talk than action, I see vendors circling around looking to flog freshly-buzzword-compliant BI tools, prognosticators trying to push nervous businesses into engaging in an arms race over data.
Of course there's real value there too, for some at least. I hope my concerns prove unfounded, but worth retaining a healthy skepticism I feel :-)
Re: Big Data's Big Problem: Little Talent
#8Actually that is silly -- McKensey should now that there is and will never be a talent shortage. There will only be shortage of talent at a particular wage rate. If the companies paid newly graduated 'data-scientists' (what other kind of scientists are there? The tea-leaf reading kind?) 200k/year then they would have a lot more. It is pretty simple economics.
Isn't that point of view also colored by ideology? Even supposing there are enough people capable of becoming that kind of "talent", what if those talents are also sought for in other kinds of jobs? Granted, if it were really urgent, perhaps companies would start looking in the most remote places for talents, so with a population of 6 billion perhaps there really would be enough who could be trained. How many of thos…
It's not that you can solve every problem, but supply and demand are very real. If the wage rates for Big Data get high enough more and more people will try out the field, which will generate a larger supply.
The reasons companies don't just throw out huge salaries though has to do with the demand side. The salaries companies are willing to pay is related to the marginal advantage they can gain from hiring someone with that skillset. If for example a company will gain say 200k per year in total advantage, that would place a hard cap on how much they would be willing to pay in salary.
So if the advantage is very high, companies will pay more. If the supply increases sufficiently wage rates will drop because there is over supply. If the supply doesn't increase enough, wages will increase more - however each company will drop out at it's own value point. This provides the natural limit to where most salaries cap out.
Re: Big Data's Big Problem: Little Talent
#9Actually that is silly -- McKensey should now that there is and will never be a talent shortage. There will only be shortage of talent at a particular wage rate. If the companies paid newly graduated 'data-scientists' (what other kind of scientists are there? The tea-leaf reading kind?) 200k/year then they would have a lot more. It is pretty simple economics.
You're absolutely right. That said, I don't buy that big-data is as revolutionary as the Internet. While in theory, every single business can collect data and optimize based on what they see, this is way too complex for most businesses to deal with. While big data has certainly been critical for the business model of ad-based startups, I don't see it being used in other industries. People keep alluding to data-driven…
This technology will change the world more than the Internet or any other technology in human history.
You're right that adoption is very slow. I'm convinced that businesses could save trillions of dollars by applying existing weak AI to their problems. Why aren't they doing it ? For one thing there's a huge gulf between the average business person's understanding of what is possible and what actually is. On the other hand the people who understand the technology don't have domain experience in various businesses. You can't develop solutions if you don't know what the problems are and it's very hard to guess at what economically relevant problems exist in fields you've never worked in.
There are probably other barriers as well. Domain experts are unlikely to champion technologies that may, well, replace them. Bayesian networks were developed in academia 20 years ago that outperformed doctors at medical diagnosis. Why aren't they being applied ? There are probably many reasons but I suspect conscious or unconscious resistance on the part of the medical community plays a significant role.
As for a talent shortage, I don't buy it. I'm exactly the sort of person this article talks about, with a strong mathematical background, excellent implementation skills and real world experience in developing and applying machine learning algorithms that have made millions of dollars for my former employers. I have had a website and a LinkedIn profile for over a year that make this fairly clear. How many consulting inquiries have I had ? Exactly zero.
Re: Big Data's Big Problem: Little Talent
#10I'm not sure that the kinds of employees that this article describes will ever be a large number. There could be more of them in the future, but someone who is top-notch at all of statistics, programming, and data-presentation has long been less common than someone who's good at one or two of those. Companies might consider looking at better ways to build teams that combine talent that exists, instead of pining for m…
I completely take most of your points, but I think that pretty much all quantitative PhD's are going to be close to "data scientists". Given that stats and explaining your research are requirements, all that's left is to train them to program, which a lot of people are already doing. As a matter of fact, since I heard about this big data stuff I've been honing my skills in this area, in case the hype actually manifes…
Anyone who can earn a PhD can learn to program but being good at engineering the very complex processes that are needed for effective machine learning applications is a skill that is not so easily acquired.
I've worked with quite a number of quantitative PhD level people in my career and most often the quality of their code leaves much to be desired.