Launching our Data Science and Big Data Track

99% of people looking for information about big data and 99% of people looking to do data science, don't have nowhere near big data, and don't need to be taught hadoop. Those people are instead often lacking fundamental knowledge and are looking for a trick technological solution instead of reviewing their basics.

A "data science" track should hence be 90% about algorithms, data structures, linear algebra and computer architecture. People need to know how to compute with matrices, how to use B-trees, what R-trees are, what SIMD is, what cache locality is, what package to use for practical linear algebra, what the gpu is good for and when and how to use it. Teach databases, but include also some of the theoretical database stuff, and teach how to correctly use relational databases in the first place, since this is most commonly useful and people don't really understand it. PostgreSQL is a real pearl, and few people know how much capability it has, how to use the geospatial indexing, the full-text search, how to do basic optimization and profiling.

Even for people who really do large scale computations, learning hadoop or mongodb (is anyone legitimately doing big data really using mongodb?) is just an afterthought, considering how much you have to learn first about mathematics, algorithms and computing to do anything sensible at that scale at all. If you bubble sort, MongoDB won't save you. If you know the fundamentals already, you likely don't need a separate course in mongo or hadoop.

For people looking to learn something more genuine, I would recommend, for example, this book:

http://infolab.stanford.edu/~ullman/mmds.html

(Course author here) Thanks. We chose Python because it's a little more approachable for many people that Java, and is the language used in Udacity's Comp Sci 101 course. Also, using Hadoop Streaming saved us from having to explain a bunch of concepts such as WritableComparables, InputFormats etc that would just have got in the way of the basic MapReduce principles.

I'm working through this course (and took the earlier sister course Computation for Data Analysis with R). It's quite good so far. They stay away from the "big" part and focus on the core of data analysis: how to find data, clean it up, explore it, find relationships and present your findings. We are using R, which is suitable for most data sizes. It's offered by Johns Hopkins, and has more of an academic bent than an industry one. Great general purpose knowledge that I think you would want before you start messing around with Hadoop.

True. You're much more likely to apply data management fundamentals in a project than optimize impala queries on petabytes of user data. If you're a 10-50 person startup...maybe even a 200 person startup, what core/critical internal problems can you think of that would require such large scale computing? Would you allocate precious resources to a 'big data' team to monitor your logs or user activity? You most likely wouldn't need to. For the most part, only the big companies deal with that much data and only a handful of people would be in charge of managing it.

Edit: I also don't want to sound close minded or rule out an era where every company, large or small, will have TB's of data on their hands. I just haven't seen any indications that we're going in that direction.

What really annoys me is this particular blog post, not just the existence of the course, for example this thing:

“What is Big Data?” They will teach you fundamental principles of Hadoop, MapReduce, and how to make sense of big data. Developers will learn skills that provide fundamental building blocks towards deriving maximum value from the world's data. Technologists and business managers will gain the knowledge to build a big data strategy around Hadoop.

In my experience, to be successful in engineering in general, one has to learn whole design spaces instead of just individual technologies. This means taking a programming languages course instead of another C++ course, taking a distributed systems course instead of taking a Hadoop course, taking a databases course instead of a MySQL course and so forth. You of course have to fiddle around with the various tools as well, but you don't need a course or an instructor for that, otherwise you often end up just following written or spoken instruction which configuration file to edit, what command or query to type, etc., which is actually much _worse_ than self-directed learning. Once you have this theoretical background and bits of varied practical experience, you can do mature decisions about which tool to pick for a particular job.

So, I would really, really like to avoid anyone I might have a chance of working with learning about "how to build a big data strategy around XXX", whatever product XXX is. You don't build anything "around" up-front assumed technologies. This is just pumping the "big data" bubble, which is certainly good for Cloudera, which makes a living based on that, but doesn't exactly sound like teaching people to make informed technical judgements. I am also not to partial to the stance that learning Hadoop and MapReduce makes you a big data expert (they offer a certificate in big data after completing the course).

And no, there is nowhere near enough algorithms courses at all. The things taught in most undergradute algorithms courses are often not really the things you need for practical large scale data processing. I posted the Jeff Ullman book precisely as an example of how good courses in handling data might look like. This material is taught very rarely.