Big Data's Big Problem: Little Talent

Surprised Ben Rooney did not mention IBM acquisition of Vivisimo the day before (Apri. 25) this article (Apr.26) http://www-03.ibm.com/press/us/en/pressrelease/37491.wss "IBM Advances Big Data Analytics with Acquisition of Vivisimo"

Basically a solution looking for a problem.

They are right, the complexity that big data caters for requires expertise at both technical and business level that would be costly (though may not be at infrastructure level). In the current economy, it looks even more difficult where businesses want to squeeze the maximum out of dollar investment.

IMO, its too early stage for big data solution adoption. However stage could be set for startups who can come up innovative solution that brings the cost level down together with simple and useful easy to grasp solutions.

As someone in the big data field on the ground (VP of Engineering). Let me give you my thoughts on it.

Your impression about the hype is correct. There are a lot of vendors offering BIG solutions, if you pay them BIGGER money. Where I used to translate the word enterprise to $$, now I translate Big Data to $$$$$$$.

When I'm hiring, I don't go looking for Big Data people, because generally they don't exist. Statistics is a really great general addition to a programmers toolkit. Machine Learning is valuable as well, although in my experience the application is more limited. What this article doesn't mention is a whole host of other skills required.

Modeling, and not just a formal mathematical model, but applying any type of model to your data to get insight. Check out the model-thinking class on coursera.

Exploratory Data Analysis, much different skill than confirmatory statistics.

Design of Experiments, specialized subfield within statistics.

Logistics, how to setup, maintain, and maximally utilize an efficient distributed cluster and build a pipeline getting your data to the cluster, cleaning it, building it into a model, and then extracting insight and delivering that end value.

Those are a couple of the skills at a high level. At a more nuts and bolts level, Hadoop is the defacto standard for Big Data. Learning how to build a data pipeline out of the Linux tool chain is very common in the data science world.

The overall value stream for Big Data is deep and wide. Most companies don't have expertise in much of these, and so at the current time you have to learn them yourself or find a company focused on building a team around it.

If you are just learning this yourself, you'll probably get an academic knowledge. If you want to make yourself valuable in the marketplace, you'll really want to get hands on experience. Knowing a z-score is one thing, building a process to gather data and compute a model against it is a whole different ball game. As the article mentions, if you have nice clean data it's easy to apply a model. If you have messy ugly data from 20 different vendors and 200 clients with various failures, anomalies, and you have to figure out what type of model is helpful, oh and you have a deadline because for 500th time someone promised something impossible to the client, then you have something closer to what Big Data is today.

* grammar edits

I agree with Paul's comment. The 250 number feels low to me, but that is applying a specific model. Typically people come with some set of favorite models, and many of them provide that vast majority of the benefit a business needs. Especially when the current model in use is slipshod and busted at best.

I see this in software dev consulting, and it's probably in many other fields as well too.

I see companies looking for one person who's a highly-skilled DBA, sysadmin, developer and who can interact affably with all levels of people in a company, including customers, at the drop of a hat. Often it's because they had one 'magical' person who did all that, though usually not very well, and the next person (or team) who comes in after on that project has to deal with a bundle of undocumented crap that never really 'worked', but worked well enough to keep some people happy.

This scenario has been surprisingly common, and I worked with a company last year who was committed enough to hire dedicated dba and sysadmin positions, vs continuing to rely on app devs to handle all that stuff. The separation of concerns has worked out pretty well, though at first there was some concern about the cost of adding 'dedicated' people. In reality, all that work was being done anyway, often in ways that weren't terribly understandable by anyone outside the project.

The time it takes to get feature X done, tested, db upgraded, rolled out, servers maintained, patches applied, etc... is going to be roughly the same. If that's going to take, say, 100 hours, it's either 3 weeks of one person, or 1 week of 3 people. It's not worked out quite that cut-and-dried, but it's coming close. And the ability for each person/team to focus on their core skills and let someone else handle the "other stuff" has meant that, generally, the quality of things is better all around.