Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead

motherduck.com

Post reply on HN
Print view View original

71–80 of 444 posts

Re: Big data is dead

#71

Post by mytherin »

glogla wrote: post #20

I agree with a lot of the sentiments of the MotherDuck people, but boy are they loud and proud for someone who never delivered anything more than blogposts and vague promise to somehow exploit the MIT licensed DuckDB. Meanwhile for example boilingdata.com seems to have already done that - by using AWS Lambda + DuckDB as distributed compute engine which I can't decide if its awesome, deranged or both.

We (the DuckDB team) are very happy working together with MotherDuck in a close partnership [1].

[1] https://duckdblabs.com/news/2022/11/15/motherduck-partnershi...

Re: Big data is dead

#72

Post by mritchie712 »

fdgsdfogijq wrote: post #14

"For more than a decade now, the fact that people have a hard time gaining actionable insights from their data has been blamed on its size." The real issue is that business people usually ignore what the data says. Wading through data takes a huge amount of thought, which is in short supply. Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven". Most corpo…

Size isn't the real problem, it's time.

Are you going to take the time / money to set up a warehouse, get all the data into with an ETL product, set up dbt or some other transformation layer, set up a BI tool and build the reports and dashboards, etc.

Regardless the size of your data, you still need to get it in one place and model it in a way it's actually usable.

Re: Big data is dead

#73

Post by AnIdiotOnTheNet »

eska wrote: post #66
slt2021 wrote: post #36

Earlier quoted context omitted.

I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick so…

This would match what psychologists say about humans in general: we feel first, then we use our brain to justify that feeling. We’re not rational beings.

We totally are, it's just that rationality is a tool, not a guide. If you want to work out the truth, rationality will help you do that, but if instead you want to justify a decision you already made, well, it'll help you do that too.

Re: Big data is dead

#74

Post by luckydata »

It's kinda weird to read this. The whole argument is "we didn't have databases that could handle the sizes and use cases emerging, we worked on the problem for 20 years and now it's no biggie".

Mission accomplished more than big data is dead IMHO.

Re: Big data is dead

#75

Post by ethbr0 »

pphysch wrote: post #37
fuzzylightbulb wrote: post #32

Earlier quoted context omitted.

"customer obsession" was always at the mercy of the real obsession: "making money hand over fist". The former will ALWAYS lose out to the latter given enough cycles.

Yeah, "customer obsession" really just means "market share / growth obsession" which is a means to (eventually) making monopoly profits. Which Amazon seems to have achieved.

One may follow the other, but not vice versus.

It's a pretty strong argument to say that Microsoft under Gates was technically obsessed, but that really faltered under Balmer.

Microsoft continued to win profits, but they made major strategic missteps that cost the revenue.

Amazon feels like it's going down the same path: empowering the tree-gazers without remembering that the forest also matters.

Re: Big data is dead

#76

Post by chongli »

slt2021 wrote: post #36
fdgsdfogijq wrote: post #14

"For more than a decade now, the fact that people have a hard time gaining actionable insights from their data has been blamed on its size." The real issue is that business people usually ignore what the data says. Wading through data takes a huge amount of thought, which is in short supply. Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven". Most corpo…

I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick so…

Same goes for economists and the politicians who sponsor them, just as it did for the astrologers and their patron kings.

Re: Big data is dead

#77

Post by pier25 »

> Are you in the big data one percent?

Exactly, and I'd go further.

Are you in the perf/scale/data one percent?

So many people worry about scaling when in reality 99% of web apps will never reach above 100reqs/s.

I've been in web dev for 20+ years. Only once when working for a big international corporate client I had to worry about traffic spikes. And that was just for one of their multiple web apps.

Re: Big data is dead

#78

Post by slt2021 »

zeagle wrote: post #67
slt2021 wrote: post #36

Earlier quoted context omitted.

I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick so…

Huh. I've not thought of it as laundering, but I think you've basically summarised consulting in healthcare. Pay to legitimize and push through a pre-existing idea (eg let's close down a few ERs) or a delusion (e.g. lean, we don't need a waiting room) and say it was recommended by consultants to stakeholders and the public.

all consulting is like that, Partners/MDs at consulting companies meet with Board/CEOs to get rough idea of what they want/need, and quickly negotiates a consulting engagement contract to create PowerPoint with all the evidence and analysis gathered that supports CEO's initial idea.

This is the only reason why a 60+ PowerPoint slide deck can cost several millions dollars

Re: Big data is dead

#79

Post by articsputnik »

I love DuckDB's simplicity and think it will solve many problems. Still, transitioning from a local single file DB to concurrent updates and serving it online will be different. I'm curious about what MotherDuck will come up with to solve DuckDB at scale.

I love use cases like the Rill Data (https://youtube.com/watch?v=XvP2-dJ4nVM), where you can suddenly run analytics with a single cmd line prompt and see your data just instantly visualized. Such use cases are only possible because of the "small" data approach that DuckDB tries.

Re: Big data is dead

#80

Post by low_tech_punk »

Long live Big Model, I guess? Instead of independent data warehouses, we are now moving towards a few centralized companies using supercomputer in physical data centers. The "winner takes all" effect will only increase as the trend goes on.
Post reply on HN