Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead

motherduck.com

Post reply on HN
Print view View original

21–30 of 444 posts

Re: Big data is dead

#21

Post by gardenhedge »

guardiangod wrote: post #4

There is literally a post on front page on ChatGPT, and Microsoft and Google are preparing to duke it out starting in the _next 2 days_ over big-data generated 'chat' result. Big data was never going to be useful to even medium size enterprises, unless anyone can get public access to PBs of data, but that doesn't mean big data is dead. ChatGPT is literally changing how school will test their students, for a start. Ma…

That is what the author said. From the article: "Big Data is real, but most people may not need to worry about it."

Re: Big data is dead

#22

Post by travisgriggs »

I've made anecdotal observations similiar to this over the last 10 years. I work in AgTech. A big push for a while here has been "more and more more data". Sensor-the-heck out of your farm, and We'll Tell You Things(tm).

Most of what we as an industry are able to tell growers is stuff they already know or suspect. There is the occasional suprise or "Aha" moment where some correlation becomes apparent, but the thing about these is that once they've been observed and understood, the value of ongoing observation drops rapidly.

A great example of this is soil moisture sensors. Every farmer that puts these in goes geek-crazy for the first year or so. It's so cool to see charts that illustrate the effect of their irrigation efforts. They may even learn a little and make some adjustments. But once those adjustments and knowledge have been applied, it's not like they really need this ongoing telementry as much anymore. They'll check periodically (maybe) to continue to validate their new assumptions, but 3 years later, the probes are often forgotten and left to rot, or reduced in count.

Re: Big data is dead

#23

Post by humanizersequel »

miguelazo wrote: post #11
guardiangod wrote: post #4

There is literally a post on front page on ChatGPT, and Microsoft and Google are preparing to duke it out starting in the _next 2 days_ over big-data generated 'chat' result. Big data was never going to be useful to even medium size enterprises, unless anyone can get public access to PBs of data, but that doesn't mean big data is dead. ChatGPT is literally changing how school will test their students, for a start. Ma…

>ChatGPT is literally changing how school will test their students, for a start. Sure, instead of schools checking for plagiarism from other students' papers using turnitin.com, they'll check for plagiarism using ChatGPT tools that scan for known output from their industrial-scale amalgamation of plagiarized materials. Big whoop.

All math homework through the high school level is now as simple as figuring out how to describe it to ChatGPT (or maybe ChatGPT 2.0 for particularly tricky examples). Paper-writing is now a matter of figuring out how to rephrase LLM output in your own words to get around any watermarking or pattern detection.

Re: Big data is dead

#24

Post by barathr »

travisgriggs wrote: post #22

I've made anecdotal observations similiar to this over the last 10 years. I work in AgTech. A big push for a while here has been "more and more more data". Sensor-the-heck out of your farm, and We'll Tell You Things(tm). Most of what we as an industry are able to tell growers is stuff they already know or suspect. There is the occasional suprise or "Aha" moment where some correlation becomes apparent, but the thing a…

Classic paper on soil moisture sensors (from 2010!) -- the title says it all:

"Mate, we don't need a chip to tell us the soil's dry"

https://doi.org/10.1145/1858171.1858211

Re: Big data is dead

#25

Post by bhhaskin »

miguelazo wrote: post #11
guardiangod wrote: post #4

There is literally a post on front page on ChatGPT, and Microsoft and Google are preparing to duke it out starting in the _next 2 days_ over big-data generated 'chat' result. Big data was never going to be useful to even medium size enterprises, unless anyone can get public access to PBs of data, but that doesn't mean big data is dead. ChatGPT is literally changing how school will test their students, for a start. Ma…

>ChatGPT is literally changing how school will test their students, for a start. Sure, instead of schools checking for plagiarism from other students' papers using turnitin.com, they'll check for plagiarism using ChatGPT tools that scan for known output from their industrial-scale amalgamation of plagiarized materials. Big whoop.

Or just require students to use software that keeps track of version history or require spyware installed.

Re: Big data is dead

#26

Post by aintgonnatakeit »

capableweb wrote: post #18
fdgsdfogijq wrote: post #14

"For more than a decade now, the fact that people have a hard time gaining actionable insights from their data has been blamed on its size." The real issue is that business people usually ignore what the data says. Wading through data takes a huge amount of thought, which is in short supply. Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven". Most corpo…

Is that what's happening at Amazon as well? As they seemingly is loosing more and more track of the "Customer Obsession" schtick.

They are encouraging their customers to have a bias toward action. Away from that asshole Bezos.

Re: Big data is dead

#27

Post by juujian »

> Most data is rarely queried

Right on point. In the past I have been obsessed with big data, looking for insights. Then I realized that a medium-sized specific data set is always better than a gargantuan general big data monster. There is so many applications in my field where only outliers matter anyways, and everything is very "centralized" to a few relevant observations. So the only thing about big data is that you maybe throw away 99.9% of the data right away and then you have some observations that you actually care about. There is soooo much data out there that is just noise, and so little that I actually care about. And that's why I still end up hand collecting stuff every now and then.

Re: Big data is dead

#28

Post by donretag »

My personal definition of Big Data has always been when you gather/store data without having a planned use for it. Do we need this data? Don't know, let's just store it for now.

The article does allude to this definition when it states that "Most data is rarely queried". We have become data hoarders. Technology has made it easy (and relatively cheap) to store data, but the ideas of what to do with this data have not scaled in comparison.

Re: Big data is dead

#29

Post by eganist »

humanizersequel wrote: post #23
miguelazo wrote: post #11

Earlier quoted context omitted.

>ChatGPT is literally changing how school will test their students, for a start. Sure, instead of schools checking for plagiarism from other students' papers using turnitin.com, they'll check for plagiarism using ChatGPT tools that scan for known output from their industrial-scale amalgamation of plagiarized materials. Big whoop.

All math homework through the high school level is now as simple as figuring out how to describe it to ChatGPT (or maybe ChatGPT 2.0 for particularly tricky examples). Paper-writing is now a matter of figuring out how to rephrase LLM output in your own words to get around any watermarking or pattern detection.

Wolfram alpha has been around for math cheats (and people like me who just needed a more visual representation to learn) for a while now. Including proof of work.

Re: Big data is dead

#30

Post by Loughla »

miguelazo wrote: post #11
guardiangod wrote: post #4

There is literally a post on front page on ChatGPT, and Microsoft and Google are preparing to duke it out starting in the _next 2 days_ over big-data generated 'chat' result. Big data was never going to be useful to even medium size enterprises, unless anyone can get public access to PBs of data, but that doesn't mean big data is dead. ChatGPT is literally changing how school will test their students, for a start. Ma…

>ChatGPT is literally changing how school will test their students, for a start. Sure, instead of schools checking for plagiarism from other students' papers using turnitin.com, they'll check for plagiarism using ChatGPT tools that scan for known output from their industrial-scale amalgamation of plagiarized materials. Big whoop.

I mean, that's certainly part of it. But you're also seeing very fast restructuring of individual courses (because programs overall will definitely take years to restructure because higher education moves so damned slow) to account for these tools.

In the small institution I am currently working with, the English courses, in one week, integrated chatgpt as a tool for students to work with. It's part of the collaborative idea building and development process now for every student enrolled in creative writing and writing analysis classes, and that happened in one week. I cannot stress enough how unbelievably fast that is for higher ed. That's faster than light speed.

And we're not even that well resourced. I have to imagine there are other examples where it's more than just running through a bot to scan for known outputs.

Post reply on HN