Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Most data isn’t “big,” and businesses are wasting money pretending it is

qz.com

Post reply on HN
Print view View original

71–80 of 160 posts

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#71

Post by biot »

Twisol wrote: post #29
Tobani wrote: post #9

Sure if you're dealing with 1GB of data it probably isn't worth spinning up a Hadoop cluster to run your analysis. However, if you already have Hadoop up an running for something that genuinely requires it, that 1GB job might make sense there. The data may already be in HDFS, and you already have the infrastructure there to manage and monitor jobs. The references to Facebook & Yahoo running small jobs on huge cluster…

"Buzzard" isn't an eggcorn I've ever heard before! Did you mean "buzz word"?

Perhaps he packs a lot of data in his carrion luggage.

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#72

Post by joseph_cooney »

fusiongyro wrote: post #62
joseph_cooney wrote: post #56

Earlier quoted context omitted.

But if google has only existed for 15 years how can the business model of "get money out of people trying to help them do a google me-too" have existed for 30?

You're putting us to sleep with this tiresome over-literal arguing. Be more interesting.

I guess as a software engineer I can be a little pedantic, literal and detail oriented, perhaps to my own detriment.

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#73

Post by hso9791 »

kamaal wrote: post #50

As some one who is currently dealing with these sort of things I can tell this article hits the nail on its head. Most, heck something like 99.99% of all so-called big data I've dealt is something I wouldn't even classify as small data. I've seen data feeds in KB's sent over to be handled in as big data. It happens all the time. A simple data problem sufficient enough to be easily solved on something like a small db…

Agreed. It is quite amazing what modern ANSI SQL has built into it. If one picks the right storage engine, an appropriate schema design and is rigorous about data quality, the flexibility and performance is incredible.

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#74

Post by xtracto »

christopheraden wrote: post #30
philip1209 wrote: post #11

I think that big data has made math sexy, and selling applied statistics and operations research to small and medium-sized businesses under the guise of "big data" with the intention of providing applied mathematical tools is what is happening in the market.

Statistics involves checking modeling assumptions. A lot of what I've seen with the big data people is the repetition of algorithms to the exclusion of understanding and checking modeling assumptions. While it's nice that the big data craze is making statistics more popular in the mainstream press, it is important that statistics does not become just an application of numerical methods without consideration of underl…

Reminds me of a quote I read somewhere about simulation models (specifically referring to Agent Based Modelling) saying that (heavily paraphrasing): "A lot of models are great random number generators"... or "garbage in, garbage out".

I suspect a lot of these people doing "big data models" are as you say, ignoring the importance of having solid assumptions.

Oh well, that's exactly in part what brought down the financial collapse: A bunch of kids get a formula (Blach-Scholes) and believe blindly in its magic powers so they apply it to everything. Fast forward several years and we've got what everybody knows.

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#75

Post by Tossrock »

Sometimes though, you really do have lots of data and need appropriate solutions. At Quantcast, our cluster processes petabytes per day and our edge datacenters handle hundreds of thousands of transactions per second. In fact we recently open sourced our file system (QFS[1]) built on top of HDFS, which can up to double FS capacity on the same hardware. Although it's certainly true that not every company (or even not most) needs all that horsepower, there are definitely some for whom it's the core of their business.

[1]. http://quantcast.github.io/qfs/

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#77

Post by rmrfrmrf »

mattzito wrote: post #49
aswanson wrote: post #44

Oracle writes shitty 'enterprise apps' (god I hate that phrase) that they sell to big companies, because their salesmen/women wear great attire and are good at mirroring dumb ceos/cios, like the ones that run several companies I have worked for. Will someone please end this nonsense? At what point does usability/stability/utility become factors?

This is a viewpoint that I hear a lot, mostly from people who are not in the room when these grand enterprise implementation decisions are made. While it's true that a good salesperson can make a difference in winning a deal vs. another vendor, salespeople almost never convince a company that they need a big enterprise software platform. 95% of the time, the company has already decided that the current way they do X…

> This means that instead of the organization saying, "We will adapt to off the shelf software and change our processes to better align with the way the software is designed to be used", they say, "Make your software work the way we do things".

This must be a damned if you do, damned if you don't kind of situation, because I work for a company that attempted to use a OOTB Oracle software package and ended up getting roundly criticized by every part of the company, both internally and externally.

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#78

Post by pekk »

It's precious to read through almost every post in this thread complaining about 'big data' and saying that everyone can just use a normal relational database or whatever. But 'big data' has brought markets to exploit to feed HN-type entrepreneurs, and jobs and loads of prestige for HN-type engineers - who I have never noticed to be shy about bragging on how much data is in their systems without regard to whether that data is particularly meaningful.

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#79

Post by rmrfrmrf »

kamaal wrote: post #50

As some one who is currently dealing with these sort of things I can tell this article hits the nail on its head. Most, heck something like 99.99% of all so-called big data I've dealt is something I wouldn't even classify as small data. I've seen data feeds in KB's sent over to be handled in as big data. It happens all the time. A simple data problem sufficient enough to be easily solved on something like a small db…

I really don't understand why anyone has a problem with relational databases. Once you take the time to understand how they work (by taking a class or reading a book), it's really straightforward and makes a lot of sense. Not to mention it's really fast and quite reliable.

I get that a NoSQL-ish alternative makes sense for companies that have tons of shards spanning the globe, but for the vast majority of people, a relational database serves just fine.

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#80

Post by hatu »

emiliobumachar wrote: post #17
Tobani wrote: post #9

Sure if you're dealing with 1GB of data it probably isn't worth spinning up a Hadoop cluster to run your analysis. However, if you already have Hadoop up an running for something that genuinely requires it, that 1GB job might make sense there. The data may already be in HDFS, and you already have the infrastructure there to manage and monitor jobs. The references to Facebook & Yahoo running small jobs on huge cluster…

Like that huge firetruck used to put out small fires. Cities only need them for big fires, but, if you gotta have one and keep it ready, it makes sense to deploy it every time.

But realistically 99% of fires are put out by a extinguisher or a pail of water.
Post reply on HN