61–70 of 235 posts
I worked on a project earlier this year that was envisioned as a hadoop setup. The end result was a 200loc python script that runs as a daily batch job - https://github.com/jamii/springer-recommendations
I'm tempted to setup a business selling 'medium-data' solutions, for people who think they need hadoop.
Earlier quoted context omitted.
Interesting, but it isn't really similar advice. Indeed, it sounds like absolutely terrible advice (unless there is some context that is missing).
I suspect it should have been "Who's worked with databases too big to fit entirely in memory". Because if it all fits then you don't have to worry too much about performance overheads. If it doesn't then you need to think about how you're going to avoid table-scans and the like.
However I would still strongly contest the analogy even if the database did fit in memory (the author may be off by magnitudes themselves, as in 1997 a 2GB database was a pretty substantial, unweidly thing for most people) -- doing the simple steps of putting your data in a database instantly enables enormous flexibility in the use of that data at very little cost or overhead, with better to enormously better performance than the average person is going to yield with a flat file.
This situation (Hadoop for big data), in contrast, is about throwing away a lot of flexibility, and paying a large performance price, to add big-data scale out flexibility. It is, in many ways, the opposite situation.
Back in college in '97 or so, in our databases class on the first day, the professor asked, "Who's worked with databases before?" A bunch of hands went up. "Oh, sorry, let me rephrase, who's worked with databases larger than a few dozen gigs?" Only one or two hands remained up. "If it's smaller than that, just save yourself the effort and use a flat-file instead." 15 years later and it's the same thing, plus a few or…
What horrible advice. Hope you dropped the class.
"Cod Normal Form? Never heard of it. I like my fish sticks made of haddock anyway."
Come to think of it, I've worked with guys who apparently learned everything they know about databases from that prof's database class, unfortunately.
Earlier quoted context omitted.
Interesting, but it isn't really similar advice. Indeed, it sounds like absolutely terrible advice (unless there is some context that is missing).
I suspect it should have been "Who's worked with databases too big to fit entirely in memory". Because if it all fits then you don't have to worry too much about performance overheads. If it doesn't then you need to think about how you're going to avoid table-scans and the like.
A lot of people use databases to implement persistent user state on top of stateless HTTP, even if the data itself is small enough to fit into memory.
This latter use was known even in 1997. For example, the book "Database Backed Web Sites: The Thinking Person's Guide to Web Publishing" was published on Jan. 1 of that year.
So even when that advice was offered, it was wrong.
99% of the cases I have seen where people have been working with tables that are in the 5+ TB range for analysis, there is some obvious way to compress the data that they have overlooked. Most analysts find some way to aggregate a dataset once, then do actual work on that aggregated dataset, rather than the raw data. In geospatial analytics, for example, a trillion records can be aggregated down to census blocks/bloc…
Hooray! Some sense at last. I have worked for at least 3 different employers that claimed to be using "Big Data". Only one of them was really telling the truth. All of them wanted to feel like they were doing something special. The sad thing is, they were all special, each in their own particular way, but none of what made each company magic and special had anything to do with the size of the data that they were hand…
CEOs want to run bigger companies, even if it's not best for shareholders.
Managers want to run large mini-empires, even if it's not best for the corporation.
Engineers like to use the latest tools, even when they're not best for the problems at hand.
Solving this requires well respected leadership that knows enough about the details, and which battles are worth fighting.
I'm always torn by these headlines: yes, many organizations lack the size of data required to take advantage of Hadoop. Few of the articles really bother explaining the advantages of Hadoop, and how what you're doing really moves the break-even point in terms of data size: - 3x replication: if the data needs to be retained long-term, slapping it on one hard drive isn't going to cut it. This is pretty poor justificati…
* Replication: Disks are cheap, data can go on as many drives and servers as you need. Master-slave replication is pretty damn bulletproof these days, and multi-master isn't as terrible as it was even a few years ago.
* Working set: All servers can have the entire working set. Additionally, features like foreign data wrappers in postgresql and federated tables in mysql mean that you can query your sharded databases, and still get aggregate results back.
* Data growth: SQL servers were Big Data before Big Data was cool. Terabyte storage clusters are no problem for an SQL database engine, with some entities having clusters in the petabyte range.
Databases are a very fast moving target of late, with competition from many fronts meaning that what was true even last year may not be true today. Additionally, with things like Postgresql's foreign data wrappers, you can choose the best data processing engine for the job -- keep the data on the databases for the nice bits of SQL, but still be able to throw it towards a hadoop cluster at a whim if needed. Using the right tool for the job is important, and equally important is keeping up with what tools are out there, because this is a rapidly evolving area of tech, and analyzing each job separately, instead of relying on any mantra, is what's needed to keep one flexible to keep up.
Related, it is remarkable how we developers routinely cite Knuth's advice about premature optimization to justify our decision when the shoe fits, and then turn around and flatly ignore the advice when it doesn't fit.
Selecting Hadoop before you have a specific and concrete need for it--or see that need approaching rapidly on the horizon--is in my experience often and surprisingly coupled with a disdain for other performance characteristics (because Knuth!). The developers prematurely selecting Hadoop as their data management platform will routinely be the same developers who believe it's reasonable for a web application with modest functionality to require dozens of application nodes to service concurrent request load measured in the mere thousands. The sad thing here being that application platforms and frameworks are not all that dissimilar; today, selecting something with higher performance in the application tier is relatively low-friction from a learning and effort perspective. But it's often not done. Meanwhile, selecting Hadoop on the data tier is a substantially different paradigm versus the alternative (as the article points out), so you have some debt incoming once you make that decision. And yet, this is done often enough for many of us to recognize the problem.
In my experience, for a modest web application, it's better to focus resources and effort on avoiding common and stupid mistakes that lead to performance and scale pain. Selecting Hadoop too early doesn't really do a whole lot to move the performance needle for a modest web application.
Trouble is, many web businesses are blind to the fact that they are a modest concern and not the next Facebook.