Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead

motherduck.com

Post reply on HN
Print view View original

31–40 of 444 posts

Re: Big data is dead

#31

Post by thfuran »

epicureanideal wrote: post #19
guardiangod wrote: post #4

There is literally a post on front page on ChatGPT, and Microsoft and Google are preparing to duke it out starting in the _next 2 days_ over big-data generated 'chat' result. Big data was never going to be useful to even medium size enterprises, unless anyone can get public access to PBs of data, but that doesn't mean big data is dead. ChatGPT is literally changing how school will test their students, for a start. Ma…

I wonder how long until training today’s ChatGPT will cost $1000 of AWS compute. 10 years? At that point, does it keep scaling or is there an S curve where 100x more data and compute only leads to a 2x improvement?

I think you're underestimating the scale by a few orders of magnitude.

Re: Big data is dead

#32

Post by fuzzylightbulb »

capableweb wrote: post #18
fdgsdfogijq wrote: post #14

"For more than a decade now, the fact that people have a hard time gaining actionable insights from their data has been blamed on its size." The real issue is that business people usually ignore what the data says. Wading through data takes a huge amount of thought, which is in short supply. Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven". Most corpo…

Is that what's happening at Amazon as well? As they seemingly is loosing more and more track of the "Customer Obsession" schtick.

"customer obsession" was always at the mercy of the real obsession: "making money hand over fist". The former will ALWAYS lose out to the latter given enough cycles.

Re: Big data is dead

#33

Post by ankrgyl »

I love DuckDB and am cheering for MotherDuck, but I think bragging about how fast you can query small data is really no different than bragging about big data. In reality, big data's success is not about data volume. It's about enabling people to effectively collaborate on data and share a single source of truth.

I don't know much about MotherDuck's plans, but I hope they're focused on making it as easy to collaborate on "small data" as Snowflake/etc. have made it to collaborate on "big data".

Re: Big data is dead

#34

Post by frgtpsswrdlame »

humanizersequel wrote: post #23
miguelazo wrote: post #11

Earlier quoted context omitted.

>ChatGPT is literally changing how school will test their students, for a start. Sure, instead of schools checking for plagiarism from other students' papers using turnitin.com, they'll check for plagiarism using ChatGPT tools that scan for known output from their industrial-scale amalgamation of plagiarized materials. Big whoop.

All math homework through the high school level is now as simple as figuring out how to describe it to ChatGPT (or maybe ChatGPT 2.0 for particularly tricky examples). Paper-writing is now a matter of figuring out how to rephrase LLM output in your own words to get around any watermarking or pattern detection.

You can't figure out how to describe a math problem correctly to ChatGPT without already knowing the solution.

Re: Big data is dead

#35

Post by andix »

I see it all the time: people develop applications that will never ever get a database size of over 100GB and are using big data databases or distributed cloud databases. Often queries only hit a small subset of the date (one customer, one user). So you could easily fit everything into one SQL database.

Using any of the traditional SQL databases takes away a lot of complications. You can do transactions, you can query whatever you want, …

And if the database may get up to 1TB, still no problem with SQL. If exceed that, you may need a professional OPs team for your database and a few giant servers, but they should easily be able to go up to 10 TB, offload some queries to secondary servers, …

Re: Big data is dead

#36

Post by slt2021 »

fdgsdfogijq wrote: post #14

"For more than a decade now, the fact that people have a hard time gaining actionable insights from their data has been blamed on its size." The real issue is that business people usually ignore what the data says. Wading through data takes a huge amount of thought, which is in short supply. Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven". Most corpo…

I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs.

I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick some flashy metric like a percentage and run around with it.

Whenever my analysis contradicted - there was so much scrutiny in numbers, data quality, etc, and even after answering all questions and concerns - analysis would be tossed away as non-actionable/useless/etc.

if you want to succeed as a Data Scientist and be praised by management - you got to provide data analysis that supports managements ideas (however wrong or ineffective they might be).

Data Scientist's job is to launder management's intuition using quantitative methods :)

Re: Big data is dead

#37

Post by pphysch »

fuzzylightbulb wrote: post #32
capableweb wrote: post #18

Earlier quoted context omitted.

Is that what's happening at Amazon as well? As they seemingly is loosing more and more track of the "Customer Obsession" schtick.

"customer obsession" was always at the mercy of the real obsession: "making money hand over fist". The former will ALWAYS lose out to the latter given enough cycles.

Yeah, "customer obsession" really just means "market share / growth obsession" which is a means to (eventually) making monopoly profits. Which Amazon seems to have achieved.

Re: Big data is dead

#38

Post by scottyah »

miguelazo wrote: post #11
guardiangod wrote: post #4

There is literally a post on front page on ChatGPT, and Microsoft and Google are preparing to duke it out starting in the _next 2 days_ over big-data generated 'chat' result. Big data was never going to be useful to even medium size enterprises, unless anyone can get public access to PBs of data, but that doesn't mean big data is dead. ChatGPT is literally changing how school will test their students, for a start. Ma…

>ChatGPT is literally changing how school will test their students, for a start. Sure, instead of schools checking for plagiarism from other students' papers using turnitin.com, they'll check for plagiarism using ChatGPT tools that scan for known output from their industrial-scale amalgamation of plagiarized materials. Big whoop.

> industrial-scale amalgamation of plagiarized materials

More Big Data!

Re: Big data is dead

#39

Post by pphysch »

guardiangod wrote: post #4

There is literally a post on front page on ChatGPT, and Microsoft and Google are preparing to duke it out starting in the _next 2 days_ over big-data generated 'chat' result. Big data was never going to be useful to even medium size enterprises, unless anyone can get public access to PBs of data, but that doesn't mean big data is dead. ChatGPT is literally changing how school will test their students, for a start. Ma…

OpenAI and Google are clearly in the 1% as TFA describes.

Re: Big data is dead

#40

Post by tpoacher »

slt2021 wrote: post #36
fdgsdfogijq wrote: post #14

"For more than a decade now, the fact that people have a hard time gaining actionable insights from their data has been blamed on its size." The real issue is that business people usually ignore what the data says. Wading through data takes a huge amount of thought, which is in short supply. Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven". Most corpo…

I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick so…

sadly same with academia and funding sources
Post reply on HN