Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big Data's Big Problem: Little Talent

online.wsj.com

Post reply on HN
Print view View original

51–60 of 161 posts

Re: Big Data's Big Problem: Little Talent

#51

Post by tomjen3 »

tgflynn wrote: post #31
csomar wrote: post #28

Earlier quoted context omitted.

I have had a website and a LinkedIn profile for over a year that make this fairly clear It doesn't work like that. You LinkedIn profile might easily land you any job in Software development, but not consulting. In my opinion, if you want to do consulting for big corp. you should figure out what it takes to it. An attractive website and presentation, few buzzwords, client testimonials, business cards, and the other bl…

That's definitely not my world and one reason I left big-corp in the first place. EDIT ADDED: It seems like a really broken market if buyer decisions are completely orthogonal to the product being purchased.

I hope you are right. If you are, you have identified a potentially very, very lucrative option for a start-up. Broken markets can provide you a lot of money when you fix them.

Re: Big Data's Big Problem: Little Talent

#52

Post by pnathan »

Tichy wrote: post #21
yummyfajitas wrote: post #19

Earlier quoted context omitted.

Companies already do pay close to $200k/year for entry level data scientists. (what other kind of scientists are there? The tea-leaf reading kind?) "Data scientist" refers to the guy who can set up a hadoop cluster, do statistics on TBs worth of data, derive useful conclusions and speed it up by tweaking the low level data formats or microoptimizing the calculation. The issue is rarely paying these guys an extra $20k…

How hard can it be, though? Like taking a normal CS person and making them versatile with hadoop and so on? Could it be done for 20K$?

> do statistics on TBs worth of data, derive useful conclusions

That's gonna be the hard part. Most CS people I've met flee from math and, more generically, theory.

Re: Big Data's Big Problem: Little Talent

#53

Post by Estragon »

tgflynn wrote: post #9
throwaway1979 wrote: post #4

Earlier quoted context omitted.

You're absolutely right. That said, I don't buy that big-data is as revolutionary as the Internet. While in theory, every single business can collect data and optimize based on what they see, this is way too complex for most businesses to deal with. While big data has certainly been critical for the business model of ad-based startups, I don't see it being used in other industries. People keep alluding to data-driven…

This technology will change the world more than the Internet or any other technology in human history. You're right that adoption is very slow. I'm convinced that businesses could save trillions of dollars by applying existing weak AI to their problems. Why aren't they doing it ? For one thing there's a huge gulf between the average business person's understanding of what is possible and what actually is. On the othe…

  I'm convinced that businesses could save trillions of
  dollars by applying existing weak AI to their problems.
Why are you convinced of this? Do you have any case studies with obvious broad applications?

  I have had a website and a LinkedIn profile for over a year that make
  this fairly clear. How many consulting inquiries have I had ? Exactly
  zero.
Neither LinkedIn nor your ISP are responsible for marketing you. Have you been presenting your software at restaurant conferences? Do you have any case studies where it saved someone x% of their food costs?

Re: Big Data's Big Problem: Little Talent

#54

Post by ambiate »

I just listened to a lecture on this at BU. Emerging Internet Technologies at IBM or something of that nature. He was basically trying to sell us his product that crawled the internet (mainly a firehose at Twitter) and gathered statistics for advertisers and presented it in pretty graphics.

The main issue they had was developing language recognition. Deciding if a user 'liked', 'loved', 'hated' or was 'neutral' about a product. Another issue that stood out to me had to do with their reliance on the internet. Just because 200 users tweeted that 'this movie is going to suck' does not really represent the overall opinion.

To reiterate, the whole buzz of the lecture was the biggest turn off. He wasn't explaining about how to expand on his product or where to go from here. Just that they had developed a product and we could use it instead of attempting to develop one ourselves.

Re: Big Data's Big Problem: Little Talent

#55

Post by OzzyB »

bmelton wrote: post #43
mgkimsal wrote: post #20

Earlier quoted context omitted.

I see this in software dev consulting, and it's probably in many other fields as well too. I see companies looking for one person who's a highly-skilled DBA, sysadmin, developer and who can interact affably with all levels of people in a company, including customers, at the drop of a hat. Often it's because they had one 'magical' person who did all that, though usually not very well, and the next person (or team) who…

Perhaps slightly offtopic, but at least in Software dev consulting, there's a market for the multi-hatted individual in consulting directly, and pay is generally commensurate with the number of hats you can speak intelligently on to a customer. This isn't necessarily true everywhere -- when I lived in Memphis, TN, I often felt that I was dooming my professional career as, every time I ran into a challenge, I'd fork m…

I don't think your off-topic at all. I think your point speaks to the core of the issue at hand, which is, companies only want to hire folks who have mastered a particular skillset that they only deem important to them at that specific point-in-time.

I don't for one minute think there aren't enough people who can't do "data driven analysis" or whatever it is that companies find important "today". Companies just don't want to invest their time and money in creating that someone who will be able to do the specific thing that they need.

They always seem to think that someone will be "out there" to solve their specific need, and all it takes is a job posting on some job board.

Take a note from the old record label industry, that would have A&R departments, who would take on an individual artist who had merit, and then nurture them to become the world-class musician that makes them a ton of money.

Re: Big Data's Big Problem: Little Talent

#57

Post by yummyfajitas »

randomStuff wrote: post #42
_delirium wrote: post #3

I'm not sure that the kinds of employees that this article describes will ever be a large number. There could be more of them in the future, but someone who is top-notch at all of statistics, programming, and data-presentation has long been less common than someone who's good at one or two of those. Companies might consider looking at better ways to build teams that combine talent that exists, instead of pining for m…

This is a total ploy by large companies to increase the H1-B visa cap. I have seen companies post job openings with starting salaries of 40K for experienced developer positions so they can then claim there were no American applicants. Its a self fulfilling prophecy, if companies outsource the jobs then people don't study those skills for fear of their job going to India; then the companies complain that there aren't…

Protecting the industry with trade barriers definitely sounds like the way to make it competitive.

If we do this our IT industry might one day lead the world, like our sugar industry, automotive manufacturing and softwood lumber do.

Re: Big Data's Big Problem: Little Talent

#58

Post by tgflynn »

tgflynn wrote: post #9
throwaway1979 wrote: post #4

Earlier quoted context omitted.

You're absolutely right. That said, I don't buy that big-data is as revolutionary as the Internet. While in theory, every single business can collect data and optimize based on what they see, this is way too complex for most businesses to deal with. While big data has certainly been critical for the business model of ad-based startups, I don't see it being used in other industries. People keep alluding to data-driven…

This technology will change the world more than the Internet or any other technology in human history. You're right that adoption is very slow. I'm convinced that businesses could save trillions of dollars by applying existing weak AI to their problems. Why aren't they doing it ? For one thing there's a huge gulf between the average business person's understanding of what is possible and what actually is. On the othe…

I couldn't reply to Estragon's comment below, so I'm replying here.

I'm convinced of it primarily because I'm convinced that the majority of activities that human workers perform do not require general intelligence (ie. strong AI). This includes most manual tasks : cleaning, cooking, customer service in restaurants, etc, but also many tasks performed by office or even "knowledge" workers.

A product my previous company developed replaced over 20K workers over a period of years. Few people even know it exists.

Once you are aware of this and follow the news you see it happening in virtually all domains: e-discovery systems replacing attorney hours, automated news story generation, etc. One area that has been lagging is robotics but this will start to develop very quickly, especially with the new DARPA Robotics Challenge.

Now to be clear many of the "Big Data" applications people are talking about may not fall into this human labor replacement category, but the underlying technologies are essentially the same.

Re: Big Data's Big Problem: Little Talent

#59

Post by tomjen3 »

enfilade wrote: post #45
maeon3 wrote: post #26

Why don't the people who hire doctors, dentists, and lawyers suffer from the same talent shortage that the people who hire 'big data' computer scientists feel? Because it's better across the board to start your own startup than work your ass off for a 4% raise at a place which recognizes you as top talent. I'm on the verge of starting a startup myself, removing myself from the people in this list. There is a shortage…

I agree with your overall line of thinking. In your last sentence you wrote: "There is a shortage of talent in computer science, but never in the other disciplines, it may take another 30 years for the suits to have the ability to understand why." Could you elaborate on this point -- do you feel that there is not a shortage in disciplines such as dentistry and law because many people are willing to work very hard for…

You can actually make good money by going into the law or medicine. You have to work and and be skilled of course, but lets be honest that is also required for a start-up.

I can't help but think that ability is because you can't run a law firm without being a lawyer so the boss has some idea of what it means to be a good lawyer and how to treat them.

Post reply on HN