Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead

motherduck.com

Post reply on HN
Print view View original

311–320 of 444 posts

Re: Big data is dead

#311

Post by mejakethomas »

nerpderp82 wrote: post #219
angry_moose wrote: post #217

Earlier quoted context omitted.

My experience with "Big Data" is it was something that couldn't be handled in a spreadsheet or on their laptop using R because it was so inefficiently coded. I got sucked into "weekly key metric takes over 14 hours to run on our multi-node kubernetes cluster" a while back. I'm not sure how many nodes it actually used, nor did I really care. Digging into it, the python code ingested about ~50GB of various files, made…

That is one of the selling points of Hadoop, you can write garbage code and scale your way out of any problem, turning the $$$ knob up to more nodes.

Snowflake too.

Inefficient sql? Crank the virtual warehouse.

Re: Big data is dead

#313

Post by juve1996 »

Mizza wrote: post #189
airstrike wrote: post #86

Earlier quoted context omitted.

> ChatGPT is literally changing how school will test their students, for a start. Here's a novel idea: test students using pen and paper?

Or, preferably, admit that testing wasn't a good idea to begin with and focus on optimizing children for learning, not test-taking.

At some point you need to see what people know to measure their progress and help them.

Re: Big data is dead

#314

Post by User23 »

slt2021 wrote: post #36
fdgsdfogijq wrote: post #14

"For more than a decade now, the fact that people have a hard time gaining actionable insights from their data has been blamed on its size." The real issue is that business people usually ignore what the data says. Wading through data takes a huge amount of thought, which is in short supply. Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven". Most corpo…

I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick so…

> I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs.

The SNAFU principle: communication is only possible between equals. When an hierarchical divide exists the subordinate will tell the superior what he wants to hear.

Re: Big data is dead

#315

Post by TimTheTinker »

SoftTalker wrote: post #56
criddell wrote: post #41

Earlier quoted context omitted.

Is it crazy to think that instead of stepping up in the war against AI we instead try to figure out a way to teach kids assuming they will use AIs?

Are we trying to produce adults who are able to think critically and creatively, and who reach their full intellectual potential, or are we trying to produce adults who can push a few buttons and blindly believe what the machine tells them?

The scene from the film Idiocracy where the main character is being triaged in a doctor's office comes to mind.

Re: Big data is dead

#316

Post by disqard »

taftster wrote: post #237

This posting was great. Highly recommended reading through. It gets really good when the author hits "Data is a Liability". > An alternate definition of Big Data is “when the cost of keeping data around is less than the cost of figuring out what to throw away.” This is exactly it. It's way too hard to go through and make decisions about what to throw away. In many respects, companies are the ultimate hoarders and can…

> if your data can fit into RAM, what exactly are we talking about Big Data for?

That's a fantastic point, and I keep mentioning the COST paper to anyone who cares:

https://www.usenix.org/system/files/conference/hotos15/hotos...

Re: Big data is dead

#317

Post by CliffStoll »

taftster wrote: post #237

This posting was great. Highly recommended reading through. It gets really good when the author hits "Data is a Liability". > An alternate definition of Big Data is “when the cost of keeping data around is less than the cost of figuring out what to throw away.” This is exactly it. It's way too hard to go through and make decisions about what to throw away. In many respects, companies are the ultimate hoarders and can…

  In a larger sense, it's a challenge to throw away stuff, just as it's difficult to trim big data.  

  As I reach retirement, our attic, bookshelves, and cabinets must be trimmed -- and each item requires attention and a decision.
  
  Some things in the attic are obvious liabilities (what to do with a mercury barometer? A radium dial pocket watch? Old electronics?)  Disposing of other stuff requires time, insight, and a sense of the future (should we keep those fingerpainted scribbles from when the kids were 3?  How about those cheesy trophies from chess club? Computer books from the 1970's? Betamax home movies? Record albums?)

Re: Big data is dead

#318

Post by sixdimensional »

It is amusing that in 2005, "VLDB" (precursor term to "big data") was defined in Wikipedia to be "larger than 1TB".. after reading through the post and the author's experience.. it would appear that this was not actually a completely terrible estimate, although there are larger and smaller: https://en.wikipedia.org/w/index.php?title=Very_large_databa...

The current version of that article states: "There is no absolute amount of data that can be cited. For example, one cannot say that any database with more than 1 TB of data is considered a VLDB. This absolute amount of data has varied over time as computer processing, storage and backup methods have become better able to handle larger amounts of data.[5] That said, VLDB issues may start to appear when 1 TB is approached,[8][9] and are more than likely to have appeared as 30 TB or so is exceeded.[10]" https://en.wikipedia.org/wiki/Very_large_database

Re: Big data is dead

#319

Post by crazygringo »

spopejoy wrote: post #283
ryadh wrote: post #277

While I get that they're sometimes useful to trigger debate, I don't really subscribe to very bold statements. We are drowning in data, it's all around us. Information overload is real. Data enables most of our daily digital experiences, from operational data to insights in the form of user facing analytics. Data systems are the backbone of the digital life. It's is an ocean and it's all about the vessel you pick to…

I guess the article title is a "bold statement" but maybe the biggest insight in there is that people don't think hard enough about throwing old data away, and it hurts them. This is a liferaft for drowning in data and is more "bold" organizationally, as it actually takes a certain kind of courage to realize you should just throw stuff away instead of succumb to the false comfort that "hey you never know when you mig…

I am baffled by this comment.

Throwing out unit tests? If you make a change and it fails a test, then you fix the bug or fix the test. I can't even imagine in what universe it's a good idea to throw away a test if it covers code in use. In what universe are unit tests "false comfort"? And if "nobody understands what's actually being tested" then you've got huge problems with your development practices.

Similarly, viewing habits from 2015 are tremendously important. There may be a show they're releasing soon that is most similar to a title released in 2015, and those stats will provide the best model. "Summarize" requires knowing how data will be used in the future, but will likely throw away what you need. Not to mention how useful and profitable vast quantities of data are for ML training.

Storing data is incredibly cheap. I'm actually curious where this desire to throw away old data comes from? I've literally never encountered it before, and it flies in the face of everything I've ever learned. The only context I know it from is data retention policies, but that's solely to limit legal liability.

Post reply on HN