Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead

motherduck.com

Post reply on HN
Print view View original

361–370 of 444 posts

Re: Big data is dead

#361

Post by dcl »

angry_moose wrote: post #217
nerpderp82 wrote: post #153

Big Data was whatever someone couldn't handle in a spreadsheet or on their laptop using R. This paper is 8 years old and it was somewhat obvious then. Scalability! But at what COST? https://www.usenix.org/system/files/conference/hotos15/hotos... A big single machine can handle 98% of peoples data reduction needs. This has always been true. Just because your laptop only has 16GB doesn't mean you need a Hadoop (or Spar…

My experience with "Big Data" is it was something that couldn't be handled in a spreadsheet or on their laptop using R because it was so inefficiently coded. I got sucked into "weekly key metric takes over 14 hours to run on our multi-node kubernetes cluster" a while back. I'm not sure how many nodes it actually used, nor did I really care. Digging into it, the python code ingested about ~50GB of various files, made…

Spending more money on compute also makes you and the team look more important, an d the problems being tackled, more challenging.

Re: Big data is dead

#362

Post by dcl »

slt2021 wrote: post #36
fdgsdfogijq wrote: post #14

"For more than a decade now, the fact that people have a hard time gaining actionable insights from their data has been blamed on its size." The real issue is that business people usually ignore what the data says. Wading through data takes a huge amount of thought, which is in short supply. Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven". Most corpo…

I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick so…

110% my experience working in data science.

I found it incredibly stressful to discover and provide analyses (even experimental results) that wasn't expected, or contradicted prior beliefs. The findings were always very harshly scrutinized, and typically lead to tons of pointless extra work to 'understand what is going on'.

Re: Big data is dead

#363

Post by andrewprock »

andrewprock wrote: post #360
fatneckbeard wrote: post #342

Earlier quoted context omitted.

yes. I used to think the Church had a honest disagreement with Galileo about heliocentricity. When I grew up I realized the Church never cared about orbits at all, what they care about is maintenance of status quo. And then when I got old, I realized, there is even a reason that some people want status quo... because they have usually been around long enough to see society fall apart into anarchy and mass murder, so…

The church was not upset about heliocentrism. They were upset that Galileo was attempting to reinterpret the words of Bible in order to bolster his astronomical authority.

"[Therefore,] when God willed that at Joshua’s command the whole system of the world should rest and should remain for many hours in the same state, it sufficed to make the sun stand still. In this manner, by the stopping of the sun, the day could be lengthened on earth—which agrees exquisitely with the literal sense of the sacred text.”

- Galileo Galilei

Re: Big data is dead

#364

Post by grugagag »

qup wrote: post #60
criddell wrote: post #41

Earlier quoted context omitted.

Is it crazy to think that instead of stepping up in the war against AI we instead try to figure out a way to teach kids assuming they will use AIs?

The average human hates changes of things they've grown used to. People are very attached to school being just like it was when they went.

As long as it pushes the students into mental effort I have no problem with changes in education. The world changes, it’s normal to have changes. But there’s nothing wrong to compare against prior models, sometimes we revisit abandoned ideas, artworks, models etc

Re: Big data is dead

#366

Post by fijiaarone »

crazygringo wrote: post #319
spopejoy wrote: post #283

Earlier quoted context omitted.

I guess the article title is a "bold statement" but maybe the biggest insight in there is that people don't think hard enough about throwing old data away, and it hurts them. This is a liferaft for drowning in data and is more "bold" organizationally, as it actually takes a certain kind of courage to realize you should just throw stuff away instead of succumb to the false comfort that "hey you never know when you mig…

I am baffled by this comment. Throwing out unit tests? If you make a change and it fails a test, then you fix the bug or fix the test. I can't even imagine in what universe it's a good idea to throw away a test if it covers code in use. In what universe are unit tests "false comfort"? And if "nobody understands what's actually being tested" then you've got huge problems with your development practices. Similarly, vie…

Unit tests are only potentially of value if the code is changing. And 90% of code never changes. And 99% of unit tests never fail. Almost all of the value of unit tests come at the time of writing (a tiny percentage of) them.

After that, they become a liability that slows down builds, makes changes brittle and code based schlerotic.

A few good unit tests are a lot better than a bunch of bad ones. And even from your statement we can tell a much more pernicious risk — the false beliefs that code coverage measures whether code is tested and that a code coverage percentage is a mark of quality or safety in its own right.

Re: Big data is dead

#367

Post by dhdgrygev »

humanizersequel wrote: post #23
miguelazo wrote: post #11

Earlier quoted context omitted.

>ChatGPT is literally changing how school will test their students, for a start. Sure, instead of schools checking for plagiarism from other students' papers using turnitin.com, they'll check for plagiarism using ChatGPT tools that scan for known output from their industrial-scale amalgamation of plagiarized materials. Big whoop.

All math homework through the high school level is now as simple as figuring out how to describe it to ChatGPT (or maybe ChatGPT 2.0 for particularly tricky examples). Paper-writing is now a matter of figuring out how to rephrase LLM output in your own words to get around any watermarking or pattern detection.

Maybe this should give you a hint that said work is completely pointless and merely exists to waste student's time.

Re: Big data is dead

#368

Post by fijiaarone »

spopejoy wrote: post #328
crazygringo wrote: post #319

Earlier quoted context omitted.

I am baffled by this comment. Throwing out unit tests? If you make a change and it fails a test, then you fix the bug or fix the test. I can't even imagine in what universe it's a good idea to throw away a test if it covers code in use. In what universe are unit tests "false comfort"? And if "nobody understands what's actually being tested" then you've got huge problems with your development practices. Similarly, vie…

The unit test story is indeed bizarre. Done right unit tests should test the unit, and you'll never hit these problems. The villians here were monstrous test fixtures instead of mocks, "testing the fixture" instead of testing the code. Both were agency trading systems so "platforms" of a sort that needed significant refactoring to mock properly, so instead tests had to inject essentially fake concrete services. Someh…

Mocks are the personification of bad data. The only meaningful measurement derived from tests with mocks is how bad the architecture is.

Re: Big data is dead

#369

Post by BLKNSLVR »

chongli wrote: post #76
slt2021 wrote: post #36

Earlier quoted context omitted.

I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick so…

Same goes for economists and the politicians who sponsor them, just as it did for the astrologers and their patron kings.

Economics is the go-to for conservative, status-quo maintaining arguments because there's a wealth of statistics and information available for "how things have always been", and precious little-to-none for "how things could be if..."

It's easier to poke holes in predictions of the future than in interpretations of the past, especially when the people making those decisions have likely reached their decision-making status through "how things have always been".

Re: Big data is dead

#370

Post by fijiaarone »

criddell wrote: post #41
miguelazo wrote: post #11

Earlier quoted context omitted.

>ChatGPT is literally changing how school will test their students, for a start. Sure, instead of schools checking for plagiarism from other students' papers using turnitin.com, they'll check for plagiarism using ChatGPT tools that scan for known output from their industrial-scale amalgamation of plagiarized materials. Big whoop.

Is it crazy to think that instead of stepping up in the war against AI we instead try to figure out a way to teach kids assuming they will use AIs?

Learn your times tables and then learn to use a calculator.

If you don’t do it that way, you’ll always be fooled by whatever the screen tells you.

Post reply on HN