Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Most data isn’t “big,” and businesses are wasting money pretending it is

qz.com

Post reply on HN
Print view View original

141–150 of 160 posts

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#141

Post by mayank »

raverbashing wrote: post #113
vasyainv wrote: post #107

Earlier quoted context omitted.

Perhaps, but can't you just organize your objects by having a table for each type and adding a column as needed? it doesn't sound like such a big deal

Looks like fun But why would I waste developer time doing that if I can only do db.table.insert(obj) - in MongoDB for example (obj is a JS object) Also, finding all objects with a field named 'field1' and value '3' is slower if you do that in a relational DB (and that's the simplest case)

Why on earth would that be slower in a relational db? That makes no sense, since indices in both nosql and relational dbs are variants of b-trees.

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#142

Post by moheeb »

mshron wrote: post #20

I've maintained for awhile now that the distinction isn't between "big" and "small" data, but between coarse and fine data. Now that everything is done through the web, previously common data sources (surveys, sales summaries, etc) are being supplanted by microdata (web logs, click logs, etc). It does take a different skill set to analyze noisy, machine-generated data than to analyze clean, survey-like data; it's a s…

I also like that distinction. To me "big" data isn't big until there is a lot of it...and there is a definite distinction between "How many bananas were sold Tuesday?" and "Was user's LED email indicator on when xyz happened?"

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#143

Post by vasyainv »

raverbashing wrote: post #113
vasyainv wrote: post #107

Earlier quoted context omitted.

Perhaps, but can't you just organize your objects by having a table for each type and adding a column as needed? it doesn't sound like such a big deal

Looks like fun But why would I waste developer time doing that if I can only do db.table.insert(obj) - in MongoDB for example (obj is a JS object) Also, finding all objects with a field named 'field1' and value '3' is slower if you do that in a relational DB (and that's the simplest case)

Well, because presumably SQL databases have features you don't get with noqsl solutions.

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#144

Post by jnazario »

scorpion032 wrote: post #127

As also some one that has been in the thick of some of the "big data" projects in the industry recently, I have to agree with the article. One of the terms I learnt in the PyData Silicon Valley in March is "Medium Data". Unless you are dealing with terabytes of RAM and Exa bytes of storage, google style, the overhead of having to maintain a cluster is something most (intelligent) people try to avoid. When you cant av…

i often tell people "if your solution to moving data necessarily involves shipping contracts" as opposed to "we'll just upload it" or even "i'll just burn it to a DVD", you're not in big data. (this is akin to "if you don't worry about power and cooling and instead worry about FLOPS, you're not in super computing" from the 90s.)

last year i was talking about an implementation we did for some data and was asked about our scale, "hundreds of terabytes" was my answer. for the people we were talking to - people who know big data - that sufficed (although a bit small on their scales, but it did require big data thinking and constructs to get answers in a reasonable amount of time).

i hadn't realized how many people were wrongly moving to "big data" solutions until i read these discussions around this article. color me surprised.

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#145

Post by raverbashing »

mayank wrote: post #141
raverbashing wrote: post #113

Earlier quoted context omitted.

Looks like fun But why would I waste developer time doing that if I can only do db.table.insert(obj) - in MongoDB for example (obj is a JS object) Also, finding all objects with a field named 'field1' and value '3' is slower if you do that in a relational DB (and that's the simplest case)

Why on earth would that be slower in a relational db? That makes no sense, since indices in both nosql and relational dbs are variants of b-trees.

Well, it's slower because of a join that exists in SQL (the relationship between your 'field/value' table and the entry. Apart from that, as you said indexes are similar.

The fun thing about NoSQL skeptics is how they think of only the current scenarios they work with, and they won't believe you until they get burned by it. So be it.

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#146

Post by [deleted] »

mayank wrote: post #141
raverbashing wrote: post #113

Earlier quoted context omitted.

Looks like fun But why would I waste developer time doing that if I can only do db.table.insert(obj) - in MongoDB for example (obj is a JS object) Also, finding all objects with a field named 'field1' and value '3' is slower if you do that in a relational DB (and that's the simplest case)

Why on earth would that be slower in a relational db? That makes no sense, since indices in both nosql and relational dbs are variants of b-trees.

[deleted]

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#147

Post by PeterisP »

jwilliams wrote: post #34

For me, "big" data is increasing the linkage between your data. It's not simply more data, but much richer, less formal data relationships. It's taking your sales data and linking it to your website clicks, linking that to the weather (or whatever). Or you take something traditionally static and add a temporal dimension. This kind of deep linking you can't measure with straight megabytes. A few gig doesn't seem that…

Well, that's the whole point - "A few gig doesn't seem that large, but if it's a complex graph with a complex hypothesis"... then it's still not 'big data'.

It's maybe smart data, maybe detailed data, but definitely not big data - that problem will have completely opposite needs and techniques than big data analysis, and should not be mischaracterised as such.

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#148

Post by jvictor118 »

mshron wrote: post #20

I've maintained for awhile now that the distinction isn't between "big" and "small" data, but between coarse and fine data. Now that everything is done through the web, previously common data sources (surveys, sales summaries, etc) are being supplanted by microdata (web logs, click logs, etc). It does take a different skill set to analyze noisy, machine-generated data than to analyze clean, survey-like data; it's a s…

I agree completely. I run a company that handles high levels of compute load for financial applications. I often describe what we do as "big compute," not big data, because the data is actually very small in size. OTOH, this tiny bit of data (real-time prices on some 1,000 assets) causes an ENORMOUS amount of computation. Often this distinction doesn't get picked up either, and people might mistakenly classify us as "big data."

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#149

Post by cma »

ctrager wrote: post #108

Even if the data isn't big, there can be a benefit from the Hadoop infrastructure. Say you have just 86,400 rows of data but each row takes 1 second. That adds up to 24 hours of elapsed time, and waiting for that run can be painful, especially if you are trying to experiment, iterate. With HDFS/MapReduce you can distribute that work across N machines and divide the elapsed time by N, speeding up the pace of iteration…

What kind of processing takes 1s per row? That's several billion instructions. And you can easily fit 86400 rows in memory, so disk seeks aren't an issue.

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#150

Post by cma »

Choronzon wrote: post #135
ims wrote: post #117

Earlier quoted context omitted.

Very true. Wouldn't the typical approach to this involve probabilistic methods like taking large-ish (but not "Big") samples from your multi TB data and doing your EDA with those?

That would work very well if our random sample accurately reflected the superset of data,which it almost always does but you also want to consider the following... Imagine our data was 98% junk with 2% of the data consisting of sequential patterns. We may be able to spot this on a graph relatively easily over the whole dataset but our random sampling would greatly reduce the quality of this information. We can extend…

Do random sampling plus n lines of surrounding context.
Post reply on HN