Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Most data isn’t “big,” and businesses are wasting money pretending it is

qz.com

Post reply on HN
Print view View original

81–90 of 160 posts

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#81

Post by zv »

Tossrock wrote: post #75

Sometimes though, you really do have lots of data and need appropriate solutions. At Quantcast, our cluster processes petabytes per day and our edge datacenters handle hundreds of thousands of transactions per second. In fact we recently open sourced our file system (QFS[1]) built on top of HDFS, which can up to double FS capacity on the same hardware. Although it's certainly true that not every company (or even not…

Thank for your self advertisment. But from what I understood, that lots of businesses threat few gigabytes as big data. It's about fashion to call yourself "big data" user.

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#82

Post by matwood »

rmrfrmrf wrote: post #79
kamaal wrote: post #50

As some one who is currently dealing with these sort of things I can tell this article hits the nail on its head. Most, heck something like 99.99% of all so-called big data I've dealt is something I wouldn't even classify as small data. I've seen data feeds in KB's sent over to be handled in as big data. It happens all the time. A simple data problem sufficient enough to be easily solved on something like a small db…

I really don't understand why anyone has a problem with relational databases. Once you take the time to understand how they work (by taking a class or reading a book), it's really straightforward and makes a lot of sense. Not to mention it's really fast and quite reliable. I get that a NoSQL-ish alternative makes sense for companies that have tons of shards spanning the globe, but for the vast majority of people, a r…

I grew up on rdbmses and think they are great. I've created numerous pieces of software backed by them. I have long preached the mantra that data is more important than the application. The data will almost always outlive the original application and become shared to others.

With that said, the above comes at a time/flexibility cost (not as bad as it used to be but still there) when building a product that isn't quite sure what it will be yet. In these cases a different data store can be beneficial since the app itself is key until traction is gained, if ever.

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#85

Post by scott_w »

achompas wrote: post #57

From the Berkeley paper on Facebook: Nonetheless, large jobs are important too. Over 80% of the IO and over 90% of cluster cycles are consumed by less than 10% of the largest jobs (7% of the largest jobs in the Facebook cluster). These large jobs, in the clusters we considered, are typically revenue-generating critical production jobs feeding front-end applications. So MR job characteristics might follow a power law…

My feeling is that the title of the article rails against companies who think they need Big Data when they don't.

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#86

Post by maggit »

rmrfrmrf wrote: post #79
kamaal wrote: post #50

As some one who is currently dealing with these sort of things I can tell this article hits the nail on its head. Most, heck something like 99.99% of all so-called big data I've dealt is something I wouldn't even classify as small data. I've seen data feeds in KB's sent over to be handled in as big data. It happens all the time. A simple data problem sufficient enough to be easily solved on something like a small db…

I really don't understand why anyone has a problem with relational databases. Once you take the time to understand how they work (by taking a class or reading a book), it's really straightforward and makes a lot of sense. Not to mention it's really fast and quite reliable. I get that a NoSQL-ish alternative makes sense for companies that have tons of shards spanning the globe, but for the vast majority of people, a r…

I don't have a problem with relational databases. Or rather, I don't have a problem with relational data modelling.

I do have a problem with Oracle. Even the Oracle experts at my former job could barely get Oracle to do something sensible, and running it on my own computer was basically a death sentence for getting anything done.

I have a problem with MySQL, from a sysadmin perspective. When I had it installed, MySQL was the package that would always break on every update. No upgrade was small enough that the data files would continue working.

(I don't have experience with Postgres, but SQLite seems more comfortable than any of the mentioned alternatives)

I have a problem with schemas in my database. It requires upfront work with modelling my data. I'd rather iterate. Also, nobody I've worked with seems to put the schemas in automatically; you need to run the special "initdb" script that isn't maintained to make it start working.

I have a problem with SQL. It would be awesome if we had a standard query language, but we don't. You can't apply the same SQL to different database engines mostly because it won't even compile, secondarily because it will give different results and finally because it will have a completely different performance profile.

All of this can be fixed by learning stuff, so I know better what I am doing.

But I already know CouchDB [1]. It took me little effort to learn, and it makes a lot of sense to my mind. I can solve problems with it, and it has neat properties with regards to master-master replication. So for me, CouchDB works just fine, just like a relational database works just fine for you :)

So, from my perspective, it seems that using some SQL solution would be the time consuming option.

[1]: CouchDB can't be considered a "Big data" database for many cases. It is slow. But it scales neatly :)

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#87

Post by burgreblast »

mshron wrote: post #20

I've maintained for awhile now that the distinction isn't between "big" and "small" data, but between coarse and fine data. Now that everything is done through the web, previously common data sources (surveys, sales summaries, etc) are being supplanted by microdata (web logs, click logs, etc). It does take a different skill set to analyze noisy, machine-generated data than to analyze clean, survey-like data; it's a s…

I like that distinction.

But I also don't really mind if Big Data is truly big, because it's clearly different data than what businesses are used to collecting and interpreting today.

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#88

Post by jackmaney »

Twisol wrote: post #29
Tobani wrote: post #9

Sure if you're dealing with 1GB of data it probably isn't worth spinning up a Hadoop cluster to run your analysis. However, if you already have Hadoop up an running for something that genuinely requires it, that 1GB job might make sense there. The data may already be in HDFS, and you already have the infrastructure there to manage and monitor jobs. The references to Facebook & Yahoo running small jobs on huge cluster…

"Buzzard" isn't an eggcorn I've ever heard before! Did you mean "buzz word"?

You have to take some of these colloquialisms with a grain assault.

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#89

Post by dasil003 »

maggit wrote: post #86
rmrfrmrf wrote: post #79

Earlier quoted context omitted.

I really don't understand why anyone has a problem with relational databases. Once you take the time to understand how they work (by taking a class or reading a book), it's really straightforward and makes a lot of sense. Not to mention it's really fast and quite reliable. I get that a NoSQL-ish alternative makes sense for companies that have tons of shards spanning the globe, but for the vast majority of people, a r…

I don't have a problem with relational databases. Or rather, I don't have a problem with relational data modelling. I do have a problem with Oracle. Even the Oracle experts at my former job could barely get Oracle to do something sensible, and running it on my own computer was basically a death sentence for getting anything done. I have a problem with MySQL, from a sysadmin perspective. When I had it installed, MySQL…

This is all pretty sane, except the schema-less part. I just don't understand why people get all hung up over schemas. Sure, migrations are a minor inconvenience, but if you just add fields in an ad-hoc fashion over time the data becomes messy and it's hard to determine any invariants about a large dataset. Sure this is avoided through careful code organization, but then aren't you just re-inventing schemas out-of-band?

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#90

Post by gizzlon »

dasil003 wrote: post #89
maggit wrote: post #86

Earlier quoted context omitted.

I don't have a problem with relational databases. Or rather, I don't have a problem with relational data modelling. I do have a problem with Oracle. Even the Oracle experts at my former job could barely get Oracle to do something sensible, and running it on my own computer was basically a death sentence for getting anything done. I have a problem with MySQL, from a sysadmin perspective. When I had it installed, MySQL…

This is all pretty sane, except the schema-less part. I just don't understand why people get all hung up over schemas. Sure, migrations are a minor inconvenience, but if you just add fields in an ad-hoc fashion over time the data becomes messy and it's hard to determine any invariants about a large dataset. Sure this is avoided through careful code organization, but then aren't you just re-inventing schemas out-of-ba…

I think this is due to how you work and what you build. If you plan it out, focus on the data structures you need and then build it, schemas are fine because you know up-front what you want.

In more ad-hoc, constantly changing, "ops this didn't work because of unknown factor X" type of projects, schemas are a pain. It's sounds really nice to have a data store that adapts when it's impossible to know up-front what you need from your data structures.

Post reply on HN