Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big Data's Big Problem: Little Talent

online.wsj.com

Post reply on HN
Print view View original

71–80 of 161 posts

Re: Big Data's Big Problem: Little Talent

#71

Post by chadmalik »

_delirium wrote: post #68
ohyes wrote: post #63

Earlier quoted context omitted.

I think another issue is that no one is interested in doing 'on the job' training. Few people learn statistics, programming, and data-presentation in college (not all three anyway). Companies might consider finding someone smart with one or two of the skills and expecting them to learn the other skills on the job. And what is talent? To me, talent is the ability to learn to do something quite well. To say there is a…

I agree somewhat, but retention is a fairly major problem. The shift away from career-length employment means that neither employers nor employees assume there will necessarily be much loyalty or longevity in the relationship. I think the decline in on-the-job training is directly related. Engineering firms used to be able to assume that it's okay to lose money on the first five years or so of an employee's work, if…

Well maybe companies should start thinking about what it takes to retain employees? Perhaps start retirement programs that incentivize people to stay with them for a number of years. As far as I can see in the economy, you are wrong. The real issue is viewing "human resources" as disposable, plug-and-play things that you use to fill stopgap needs, not some desire by employees to job-hop all the time. A lot of people, especially when they start to settle down with a mortgage and/or kids, would probably love to have employment stability.

Re: Big Data's Big Problem: Little Talent

#72

Post by ImprovedSilence »

gambler wrote: post #70

Managers frequently wail about skill shortages, but very often it's pure hypocrisy. The real problem is the reluctance to do any training (and I don't mean formal training) combined with the desire to get proven experts in whatever field. Proven experts must have years of experience in applying their expertise. If no one lets people with less experience to work in that field, where the hell would those experts appear…

True statement. It seems the trend now days is "get a grad degree, foot the bill and time yourself". Many companies I see and work with tend towards that mentality, as opposed to building a base of highly skilled workers on from the inside. It's easier and cheaper for a company to ask if you have a piece of paper, than for them to train you and get you up to speed.

Re: Big Data's Big Problem: Little Talent

#73

Post by chadmalik »

yummyfajitas wrote: post #57
randomStuff wrote: post #42

Earlier quoted context omitted.

This is a total ploy by large companies to increase the H1-B visa cap. I have seen companies post job openings with starting salaries of 40K for experienced developer positions so they can then claim there were no American applicants. Its a self fulfilling prophecy, if companies outsource the jobs then people don't study those skills for fear of their job going to India; then the companies complain that there aren't…

Protecting the industry with trade barriers definitely sounds like the way to make it competitive. If we do this our IT industry might one day lead the world, like our sugar industry, automotive manufacturing and softwood lumber do.

Despite your sarcasm, do you have any evidence of nations who have no import restrictions or government subsidies and have built up a competency? China and India are notorious for their import controls, so those are off the table for you to use. And don't use the US technology industry, which has been massively subsidized by government research, import controls, and purchasing. Hey, even the US sugar industry benefits massively from tariffs; without them it would have to compete with much lower priced product from warmer nations.

Re: Big Data's Big Problem: Little Talent

#74

Post by paulsutter »

This article is nonsense.

Talented developers are talented developers. At Quantcast we used Hadoop in production before it was even called Hadoop and now we process 10PB a day. We forbid our sourcers from using Hadoop as a resume search term because it meant absolutely nothing.

Statisticians who can code are scarce, but companies that know how to use them are scarcer.

Re: Big Data's Big Problem: Little Talent

#75

Post by why-el »

So I was wondering if any fellow HNer is on a quest to be at least comfortable around these problems. Can you share your plans? Currently I am starting with some linear algebra and I have plans to move to statistics then pick up a book on machine learning. I would really use some advice.

Re: Big Data's Big Problem: Little Talent

#76

Post by paulsutter »

gambler wrote: post #70

Managers frequently wail about skill shortages, but very often it's pure hypocrisy. The real problem is the reluctance to do any training (and I don't mean formal training) combined with the desire to get proven experts in whatever field. Proven experts must have years of experience in applying their expertise. If no one lets people with less experience to work in that field, where the hell would those experts appear…

Companies looking for "proven experts" are working on yesterday's problem. Find a better employer. Move somewhere with more choices if you need to.

Re: Big Data's Big Problem: Little Talent

#77

Post by ryanlchan »

disgruntledphd2 wrote: post #27
Dn_Ab wrote: post #11

Earlier quoted context omitted.

_delirium's post acknowledges your point but is looking for an even rarer person: "There could be more of them in the future, but someone who is top-notch at all of statistics, programming, and data-presentation has long been less common than someone who's good at one or two of those" . Someone that can program, understands statistics and can present the data in an appealing manner without losing significant fidelity…

Again, I completely see the difference (and am actually in the process of moving full time to the private sector from academia, so will probably understand a lot more in six months) but visualising data well is not that hard. Step 1: learn R Step 2: Learn PCA Step 3: Learn ggplot2 Step 4: play with the different geoms until you understand them (seriously though, everyone's eyes are optimised to find patterns, and if…

Allow a short vignette from a former academic and now management consultant.

We spent six months at a major pharmaceuticals client examining their reimbursement data. Poring over many millions of rows of transaction data and thousands of payment codes (which, of course, were unique across sales geographies), we determined the ten regions at highest risk of reimbursement collapse. R was used, maps were created, beers all around.

But almost none of it was used for the executive presentation. In fact, the only part that was included was that we had ten regions that needed fixing, and our suggestions on how to fix it. You see, the CEO was dyslexic, the chairman of the board was colorblind, and the COO was a white-boarding kind of gal, so given this audience the nuts and bolts of our advanced statistical analysis were simply irrelevant.

This is hardly surprising. If we are having so much trouble hiring people who are fluent in Big Data, how can we expect business leaders to be even conversant? With only slight exaggeration, the way you do your analysis and the visualizations that you create are not important.

Companies are demanding Big Data scientists because they suddenly have lots of data and see the term Data Scientist in the news. But what they really want is not Data Scientists, it's business insights and implications from Big Data. The customer needs 1/4" holes, but we're all arguing over which brand of laser powered diamond drill they should buy.

Re: Big Data's Big Problem: Little Talent

#78

Post by v0cab »

yummyfajitas wrote: post #57
randomStuff wrote: post #42

Earlier quoted context omitted.

This is a total ploy by large companies to increase the H1-B visa cap. I have seen companies post job openings with starting salaries of 40K for experienced developer positions so they can then claim there were no American applicants. Its a self fulfilling prophecy, if companies outsource the jobs then people don't study those skills for fear of their job going to India; then the companies complain that there aren't…

Protecting the industry with trade barriers definitely sounds like the way to make it competitive. If we do this our IT industry might one day lead the world, like our sugar industry, automotive manufacturing and softwood lumber do.

Or: Like the South Korean electronics industry does.

Re: Big Data's Big Problem: Little Talent

#79

Post by gaius »

randomStuff wrote: post #42
_delirium wrote: post #3

I'm not sure that the kinds of employees that this article describes will ever be a large number. There could be more of them in the future, but someone who is top-notch at all of statistics, programming, and data-presentation has long been less common than someone who's good at one or two of those. Companies might consider looking at better ways to build teams that combine talent that exists, instead of pining for m…

This is a total ploy by large companies to increase the H1-B visa cap. I have seen companies post job openings with starting salaries of 40K for experienced developer positions so they can then claim there were no American applicants. Its a self fulfilling prophecy, if companies outsource the jobs then people don't study those skills for fear of their job going to India; then the companies complain that there aren't…

One other thing is that the traditional entry route to software for the non-traditional candidate was via the helpdesk or QA department. You got your foot in the door, impressed the established engineers by turning around tickets quickly or by writing comprehensive bug reports, then when they needed another developer, you got the tap on the shoulder. Some of the best engineers I've worked with have come in through this route, including the best manager I've worked for.

If these kinds of positions are outsourced, how do we tap into that seam of talent anymore?

Post reply on HN