Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead

motherduck.com

Post reply on HN
Print view View original

41–50 of 444 posts

Re: Big data is dead

#41

Post by criddell »

miguelazo wrote: post #11
guardiangod wrote: post #4

There is literally a post on front page on ChatGPT, and Microsoft and Google are preparing to duke it out starting in the _next 2 days_ over big-data generated 'chat' result. Big data was never going to be useful to even medium size enterprises, unless anyone can get public access to PBs of data, but that doesn't mean big data is dead. ChatGPT is literally changing how school will test their students, for a start. Ma…

>ChatGPT is literally changing how school will test their students, for a start. Sure, instead of schools checking for plagiarism from other students' papers using turnitin.com, they'll check for plagiarism using ChatGPT tools that scan for known output from their industrial-scale amalgamation of plagiarized materials. Big whoop.

Is it crazy to think that instead of stepping up in the war against AI we instead try to figure out a way to teach kids assuming they will use AIs?

Re: Big data is dead

#42

Post by the8472 »

epicureanideal wrote: post #19
guardiangod wrote: post #4

There is literally a post on front page on ChatGPT, and Microsoft and Google are preparing to duke it out starting in the _next 2 days_ over big-data generated 'chat' result. Big data was never going to be useful to even medium size enterprises, unless anyone can get public access to PBs of data, but that doesn't mean big data is dead. ChatGPT is literally changing how school will test their students, for a start. Ma…

I wonder how long until training today’s ChatGPT will cost $1000 of AWS compute. 10 years? At that point, does it keep scaling or is there an S curve where 100x more data and compute only leads to a 2x improvement?

> At that point, does it keep scaling or is there an S curve where 100x more data and compute only leads to a 2x improvement?

Careful with the scales. 2x improvement could be interpreted from 80% human performance to 160% human performance. Or going from 10% error rate to 5% (which again crosses into superhuman territory on some tasks). Those last few bits are the critical ones.

Re: Big data is dead

#43

Post by pphysch »

epicureanideal wrote: post #19
guardiangod wrote: post #4

There is literally a post on front page on ChatGPT, and Microsoft and Google are preparing to duke it out starting in the _next 2 days_ over big-data generated 'chat' result. Big data was never going to be useful to even medium size enterprises, unless anyone can get public access to PBs of data, but that doesn't mean big data is dead. ChatGPT is literally changing how school will test their students, for a start. Ma…

I wonder how long until training today’s ChatGPT will cost $1000 of AWS compute. 10 years? At that point, does it keep scaling or is there an S curve where 100x more data and compute only leads to a 2x improvement?

We would need to see incredible advances in energy efficiency for that to happen.

Re: Big data is dead

#44

Post by tomwheeler »

danuker wrote: post #6

I agree with many of the points here. My cheap no-name old laptop SSD writes with 170MB/s. A customer has a name, address, email and order. Let's say 200 bytes for each. That means I can write 844000 new customers per second, far outside my personal marketing reach. My disk is 240GB, which means I can store data for 1.2 billion customers. It'll take a while until I become that successful.

Presumably the "order" you mention is a primary key to another table, likely one that references the individual items that make up that order, so the data will be much larger than you estimate.

It will grow larger still if you include web logs from your e-commerce site and event data from your mobile app so that you can correlate these orders with items that customers considered but ultimately didn't buy. How will your laptop and SSD perform when you then build a user-item matrix to generate product recommendations for each of those 1.2 billion customers?

While plenty of organizations unnecessarily use Big Data tools to store and analyze relatively small amounts of data, there are plenty of customers with enough data to require them. I've seen plenty of them firsthand.

Re: Big data is dead

#45

Post by bfrog »

This reminds me of a great blog post by Frank McSherry (Materialize, timely dataflow, etc) talking about how using the right tools on a laptop could beat out a bunch of these JVM distributed querying tools because... data locality basically.

https://github.com/frankmcsherry/blog/blob/master/posts/2015...

Re: Big data is dead

#46

Post by hgsgm »

revolvingocelot wrote: post #16
fdgsdfogijq wrote: post #14

"For more than a decade now, the fact that people have a hard time gaining actionable insights from their data has been blamed on its size." The real issue is that business people usually ignore what the data says. Wading through data takes a huge amount of thought, which is in short supply. Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven". Most corpo…

It's absolutely this. "Decision-based evidence-making" is what I've seen it called.

"We make decisions based on data, so let's use mine."

Re: Big data is dead

#47

Post by ren_engineer »

fdgsdfogijq wrote: post #14

"For more than a decade now, the fact that people have a hard time gaining actionable insights from their data has been blamed on its size." The real issue is that business people usually ignore what the data says. Wading through data takes a huge amount of thought, which is in short supply. Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven". Most corpo…

"data" is usually just twisted to make leadership look good or justify what decision they wanted to make anyway. Analytics data is sliced at arbitrary time periods to make growth in whatever metric look good, certain subsets are just removed, etc.

doesn't help that most of this data goes through multiple layers of BS where each person is putting it through filters to make themselves look better. And a good chunk of people don't have enough understanding of stats to understand when they are being tricked

Re: Big data is dead

#49

Post by fredliu »

The title might be hyperbole (intentionally), but the observations are more or less in line with what I experienced through a few the Big Data initiatives over the years under different enterprise environments (although I have reservation about the one 1%er comment). To me, Big Data was never about how "big" the data was, but more about the tools/system/practice needed to overcome the limitation of the previous generation. From that perspective, yes, the "monolith" may be having a "coming back" for now due to the improvement of underlying single node performance. But I do think Data size will keep growing, everything needed to make Big Data work would still be there when the pendulum swings back where a single node can't handle it anymore.

Re: Big data is dead

#50

Post by hgsgm »

Loughla wrote: post #30
miguelazo wrote: post #11

Earlier quoted context omitted.

>ChatGPT is literally changing how school will test their students, for a start. Sure, instead of schools checking for plagiarism from other students' papers using turnitin.com, they'll check for plagiarism using ChatGPT tools that scan for known output from their industrial-scale amalgamation of plagiarized materials. Big whoop.

I mean, that's certainly part of it. But you're also seeing very fast restructuring of individual courses (because programs overall will definitely take years to restructure because higher education moves so damned slow) to account for these tools. In the small institution I am currently working with, the English courses, in one week, integrated chatgpt as a tool for students to work with. It's part of the collaborat…

Wow, instead of learning and being creative, just write a out whatever the generic safe whitewashed chatbot says?

Why pay for the college?

Post reply on HN