Looks like the prelude to yet another H1-B buildup.
Big Data's Big Problem: Little Talent
41–50 of 161 posts
Re: Big Data's Big Problem: Little Talent
#42I'm not sure that the kinds of employees that this article describes will ever be a large number. There could be more of them in the future, but someone who is top-notch at all of statistics, programming, and data-presentation has long been less common than someone who's good at one or two of those. Companies might consider looking at better ways to build teams that combine talent that exists, instead of pining for m…
This is a total ploy by large companies to increase the H1-B visa cap. I have seen companies post job openings with starting salaries of 40K for experienced developer positions so they can then claim there were no American applicants.
Its a self fulfilling prophecy, if companies outsource the jobs then people don't study those skills for fear of their job going to India; then the companies complain that there aren't enough Americans with CS degrees so they need more H1-B visas.
The US needs tariffs in the IT industry to save our skill base so we can be competitive long term.
Re: Big Data's Big Problem: Little Talent
#43I'm not sure that the kinds of employees that this article describes will ever be a large number. There could be more of them in the future, but someone who is top-notch at all of statistics, programming, and data-presentation has long been less common than someone who's good at one or two of those. Companies might consider looking at better ways to build teams that combine talent that exists, instead of pining for m…
I see this in software dev consulting, and it's probably in many other fields as well too. I see companies looking for one person who's a highly-skilled DBA, sysadmin, developer and who can interact affably with all levels of people in a company, including customers, at the drop of a hat. Often it's because they had one 'magical' person who did all that, though usually not very well, and the next person (or team) who…
Perhaps slightly offtopic, but at least in Software dev consulting, there's a market for the multi-hatted individual in consulting directly, and pay is generally commensurate with the number of hats you can speak intelligently on to a customer.
This isn't necessarily true everywhere -- when I lived in Memphis, TN, I often felt that I was dooming my professional career as, every time I ran into a challenge, I'd fork my efforts and start tackling it. This means that I was getting skilled in (but not expert in) a wide variety of things. In short, my knowledge set was very wide, but somewhat shallow.
It wasn't until I moved to the DC area that I realized there's not only a market for that type of person, but a fairly lucrative one.
There are other tracks too, Architect, Management, whatever... whereas the scale repairman who can weld, machine and EE on a scale system is not exactly limited to just working on scale systems, but isn't necessarily opened up to as many positions as with software dev.
Re: Big Data's Big Problem: Little Talent
#44Earlier quoted context omitted.
deadline because for 500th time someone promised something impossible to the client This is a killer in machine learning applications. The toolsets rarely cover the entire extent of what needs to be done, so at least some custom code needs to be written. But results aren't deterministic - you don't really know if it's going to work until you run it. Several iterations are often needed to get to the first useable resu…
I couldn't agree more. Accuracy is a problem, variation is another problem. Dealing with layers in the business who have no math or statistics background but very strong opinions is yet another complication. These types of conversations aren't uncommon. Other - "I need you to prove our stuff does X, Y, and Z". Me - "Ok.." Me - "Ok the data shows our stuff does X but Y and Z are just random noise" Other - "We ran it o…
Man, this is statistics. You should be able to get any result you want!
I'm only half kidding. I can remember writing my first report (project summary) when I did a contract right after grad school. I put in maybe 5 graphs. Two looked good, three looked bad. The project manager just deleted the bad looking graphs and sent it on to the client.
Re: Big Data's Big Problem: Little Talent
#45Why don't the people who hire doctors, dentists, and lawyers suffer from the same talent shortage that the people who hire 'big data' computer scientists feel? Because it's better across the board to start your own startup than work your ass off for a 4% raise at a place which recognizes you as top talent. I'm on the verge of starting a startup myself, removing myself from the people in this list. There is a shortage…
I agree with your overall line of thinking.
In your last sentence you wrote: "There is a shortage of talent in computer science, but never in the other disciplines, it may take another 30 years for the suits to have the ability to understand why."
Could you elaborate on this point -- do you feel that there is not a shortage in disciplines such as dentistry and law because many people are willing to work very hard for only 4% raises?
Thank you!
Re: Big Data's Big Problem: Little Talent
#46Earlier quoted context omitted.
I completely take most of your points, but I think that pretty much all quantitative PhD's are going to be close to "data scientists". Given that stats and explaining your research are requirements, all that's left is to train them to program, which a lot of people are already doing. As a matter of fact, since I heard about this big data stuff I've been honing my skills in this area, in case the hype actually manifes…
_delirium's post acknowledges your point but is looking for an even rarer person: "There could be more of them in the future, but someone who is top-notch at all of statistics, programming, and data-presentation has long been less common than someone who's good at one or two of those" . Someone that can program, understands statistics and can present the data in an appealing manner without losing significant fidelity…
I still remember my first business presentation. I had a slide talking about how I did a statistics study. I was told to take the word "study" out because it had bad connotations for the target audience (middle managers at Bristol-Myers Squibb if you're curious).
The comment was probably right. But I was horrified.
Re: Big Data's Big Problem: Little Talent
#47Earlier quoted context omitted.
deadline because for 500th time someone promised something impossible to the client This is a killer in machine learning applications. The toolsets rarely cover the entire extent of what needs to be done, so at least some custom code needs to be written. But results aren't deterministic - you don't really know if it's going to work until you run it. Several iterations are often needed to get to the first useable resu…
The company I used to work for had a performance based product. They only got payed if they actually showed improved accuracy against a given evaluation set. Then they got a fraction of the cost savings (say 1 year's worth). This seems like it could be a good model for machine learning consulting, and one that I would certainly be willing to explore. It would work something like this : 1) You show me your problem and…
I think there will be a day when this will work. But right now my concerns would be:
1) The people looking for outside help probably don't have ANY model, so baselines are difficult. (I use synthetic baselines like just predicting the average of the predicted variable every time, and that's a very valuable tool, but I don't think you could get paid by beating them.)
2) When would you cut your losses on a failed project and move on? That would be incredibly difficult to do on a project that you had spent weeks or months on and not been paid. It's like cutting your losses on a failed trade in the stock market... it seems like it would be easy and obvious until you actually experience it yourself.
3) Once a company was in a position to take you up on their offer, they might as well post it on Kaggle and get a hundred people to work on it for peanuts. I'm really rooting for Kaggle b/c one of their long term goals is to let people like you (and me) make a living doing analytics work like you describe. But right now they just don't have the volume and all the projects pay out just a few thousand dollars (and only if you beat the other hundred participants).
4) If I was a company, and didn't have the expertise in house to build the model myself, I'd be wary I was really getting what I paid for. If I'm paying for a 10% boost in accuracy, how do I measure it rather than just taking your word for it?
Re: Big Data's Big Problem: Little Talent
#48Actually that is silly -- McKensey should now that there is and will never be a talent shortage. There will only be shortage of talent at a particular wage rate. If the companies paid newly graduated 'data-scientists' (what other kind of scientists are there? The tea-leaf reading kind?) 200k/year then they would have a lot more. It is pretty simple economics.
Companies already do pay close to $200k/year for entry level data scientists. (what other kind of scientists are there? The tea-leaf reading kind?) "Data scientist" refers to the guy who can set up a hadoop cluster, do statistics on TBs worth of data, derive useful conclusions and speed it up by tweaking the low level data formats or microoptimizing the calculation. The issue is rarely paying these guys an extra $20k…
They do? Which companies? How do I find them? :p
Re: Big Data's Big Problem: Little Talent
#49Earlier quoted context omitted.
This technology will change the world more than the Internet or any other technology in human history. You're right that adoption is very slow. I'm convinced that businesses could save trillions of dollars by applying existing weak AI to their problems. Why aren't they doing it ? For one thing there's a huge gulf between the average business person's understanding of what is possible and what actually is. On the othe…
Whether or not you have these skills: potential employers need to SEE them. There are lots of pretenders out there, and employers are appropriately wary. Are you showing employers results on your web page that a worse ML practitioner can't match. Putting results from a kaggle competition on my LinkedIn page landed me my current job (and I am still contacted by potential employers every couple weeks). The employers of…
Did you win the competition ? If not what was your rank, if you don't mind sharing ? I'd be really interested to know how much impact this could have.
Re: Big Data's Big Problem: Little Talent
#50Actually that is silly -- McKensey should now that there is and will never be a talent shortage. There will only be shortage of talent at a particular wage rate. If the companies paid newly graduated 'data-scientists' (what other kind of scientists are there? The tea-leaf reading kind?) 200k/year then they would have a lot more. It is pretty simple economics.
Isn't that point of view also colored by ideology? Even supposing there are enough people capable of becoming that kind of "talent", what if those talents are also sought for in other kinds of jobs? Granted, if it were really urgent, perhaps companies would start looking in the most remote places for talents, so with a population of 6 billion perhaps there really would be enough who could be trained. How many of thos…
If those talents are also sought after for other jobs then the price will go up until one of the jobs will be done by some other method or some other person. I do have trouble imagining that anybody who is working on a farm would be a good data-scientist but then I no very, very little about farming.
Economics is not tainted or colored by ideology, it is a science. It is the study of how best to allocate limited resources that have multiple conflicting uses.
In this world there is nothing that is free, everything comes with some price. As long as there is a human want that is not fulfilled, there is no additional humans.
That isn't necessarily bad though. You can charge societies progress to how few people are required to provide food to the rest. Once most Americans worked in argriculture, now only a few do. That is a good thing, because the rest of us can the do something else and satisfy some other human want.
And the remaining farmers are better of too, since they don't have to work as hard and have things like tvs and computers.