Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead

motherduck.com

Post reply on HN
Print view View original

301–310 of 444 posts

Re: Big data is dead

#301

Post by whoevercares »

slt2021 wrote: post #36
fdgsdfogijq wrote: post #14

"For more than a decade now, the fact that people have a hard time gaining actionable insights from their data has been blamed on its size." The real issue is that business people usually ignore what the data says. Wading through data takes a huge amount of thought, which is in short supply. Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven". Most corpo…

I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick so…

True - aka “are right a lot”

Re: Big data is dead

#302

Post by ryanwaggoner »

karmajunkie wrote: post #61
slt2021 wrote: post #36

Earlier quoted context omitted.

I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick so…

"Data launderer" would be a good job title...

Decision laundering. Take in dirty decisions and produce clean ones.

Re: Big data is dead

#303

Post by pron »

travisgriggs wrote: post #22

I've made anecdotal observations similiar to this over the last 10 years. I work in AgTech. A big push for a while here has been "more and more more data". Sensor-the-heck out of your farm, and We'll Tell You Things(tm). Most of what we as an industry are able to tell growers is stuff they already know or suspect. There is the occasional suprise or "Aha" moment where some correlation becomes apparent, but the thing a…

Interesting. Sounds as if what's really needed isn't so much collecting and analysing lots of data, but an alarm that's triggered when observations deviate from a set of assumptions. Observations that confirm some definition of "normalcy" -- as most observations would -- can be discarded.

Re: Big data is dead

#304

Post by threeseed »

beckingz wrote: post #278
threeseed wrote: post #248

Earlier quoted context omitted.

This is such a lazy response. I/O performance is just one of many characteristics that impact performance and from experience the one you least need to worry about. RAID 0 across multiple high-end NVME drives with OS file caching is going to be more than fast enough for most use cases. The issue is running out of CPU performance and being able to seamlessly scale up/down compute with live running workloads.

A large computer is radically CPU overprovisioned for most workloads.

But we aren't talking about most workloads.

Re: Big data is dead

#305

Post by tuatoru »

didip wrote: post #285
travisgriggs wrote: post #22

I've made anecdotal observations similiar to this over the last 10 years. I work in AgTech. A big push for a while here has been "more and more more data". Sensor-the-heck out of your farm, and We'll Tell You Things(tm). Most of what we as an industry are able to tell growers is stuff they already know or suspect. There is the occasional suprise or "Aha" moment where some correlation becomes apparent, but the thing a…

I've been telling folks, storing everything all the time is wasteful, a better alternative is: 1. Keep the raw full data for short period of time, at most 1 month. 2. Downsample what you need for longer period of time (5-10% of the full data). 3. Aggregate your metrics on a yearly basis to save money and compute costs.

Yeah, there is perhaps a data usefulness duck curve. First data useful for specific immediate problems, not a lot of use for a while after that, then 15 or 20 years later, the big picture trends start to provide value for big decision making.

Not many orgs keep their data that long, though. Or even think about the future that far.

Re: Big data is dead

#306

Post by college_physics »

Not dead, just complying with the Gartner cycle for hypes.

There is probably a rational, well thought out classification of different types of data bigness, as in CERN-big, Google-big, MegaBank-big, down to wordpress-log big and on the basis of that one would probably find that different designs are indispensable, address different pain points and cannot really "die". Hype has a more erratic lifecycle than real needs

Re: Big data is dead

#307

Post by Karellen »

eska wrote: post #66
slt2021 wrote: post #36

Earlier quoted context omitted.

I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick so…

This would match what psychologists say about humans in general: we feel first, then we use our brain to justify that feeling. We’re not rational beings.

"Man is not a rational animal; he is a rationalizing animal."

-- Robert A. Heinlein, Tunnel in the Sky, 1955

Re: Big data is dead

#308

Post by sportstuff »

fdgsdfogijq wrote: post #14

"For more than a decade now, the fact that people have a hard time gaining actionable insights from their data has been blamed on its size." The real issue is that business people usually ignore what the data says. Wading through data takes a huge amount of thought, which is in short supply. Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven". Most corpo…

No data is better than Bad data

Re: Big data is dead

#310

Post by _hcuq »

slt2021 wrote: post #36
fdgsdfogijq wrote: post #14

"For more than a decade now, the fact that people have a hard time gaining actionable insights from their data has been blamed on its size." The real issue is that business people usually ignore what the data says. Wading through data takes a huge amount of thought, which is in short supply. Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven". Most corpo…

I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick so…

So Big Data isn’t dead. It’s just found it’s place in society….
Post reply on HN