Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead

motherduck.com

Post reply on HN
Print view View original

291–300 of 444 posts

Re: Big data is dead

#291

Post by nemo44x »

tylerhannan wrote: post #223
nemo44x wrote: post #205

Why would I use DuckDB instead of Clickhouse or similar? Is it just because I want to have the database embedded in my app and not connect to a server?

One great reason to use DuckDB was when ClickHouse took up too much memory on Parquet files. https://github.com/ClickHouse/ClickHouse/issues/45741#issuec... helps with that though. Also, clickhouse-local exists https://clickhouse.com/blog/extracting-converting-querying-l... as a thing. But, yes, when I think of DuckDB...I think embedded use cases...i'm also not a power user. I also think of this very much as a 'horse…

Thanks but I’m looking for specific use cases. Like I get SQLite. And I get Clickhouse. But I just don’t get why I’d use DuckDB specifically. I’m sure it’s awesome and super useful but I have a gap in my understanding.

Re: Big data is dead

#292

Post by Loughla »

stcroixx wrote: post #259
Loughla wrote: post #150

Earlier quoted context omitted.

What a weird assumption you made there. In what way does what I wrote sound panicked? Because it was a week? Yes it was fast, but it was a massive effort of the entire english faculty. It's integrated into existing assignments, modifying processes that are super well established already. It was like integrating a new person into the class. Also, it was before semester, so the students literally saw nothing weird; tha…

I simply don’t buy that they had anywhere close to enough time to integrate this into an existing curriculum in a way that would meet the high standards paying students should have for a university education. I don’t have to ask how an English department became experts in using a just released AI in the classroom in a week because I don’t think that’s what they actually achieved.

Again, stop putting words in my mouth, please. I never said they became experts. I said they integrated it into existing processes. I said that they were doing something other than just scanning for chatgpt hits like plagiarism checkers.

>It's part of the collaborative idea building and development process now for every student enrolled in creative writing and writing analysis classes

>It's integrated into existing assignments, modifying processes that are super well established already. It was like integrating a new person into the class.

I don't honestly know how else to say that. I legitimately do not know how to help you understand what I'm saying.

Re: Big data is dead

#293

Post by cmollis »

we regularly run audits on over 12 years of customer order histories. This requires scanning of about 40TB of data and growing. They used to jump through hoops on the Oracle cluster just to get data out for one customer. We pushed all of the order history into s3 parquet using Spark and I can query this in about 20 seconds using Spark or Presto. It's now streamed through kafka and Spark structured streaming so it's up to date in about 3 minutes. The click-bait-y title notwithstanding, I get that not all data is 'big' and duckdb (and datafusion, polars, etc) is probably great for certain use-cases but what I work on every day can't be done on a single machine.

Re: Big data is dead

#294

Post by [deleted] »

gffrd wrote: post #105
travisgriggs wrote: post #22

I've made anecdotal observations similiar to this over the last 10 years. I work in AgTech. A big push for a while here has been "more and more more data". Sensor-the-heck out of your farm, and We'll Tell You Things(tm). Most of what we as an industry are able to tell growers is stuff they already know or suspect. There is the occasional suprise or "Aha" moment where some correlation becomes apparent, but the thing a…

I've observed the same in manufacturing … and fitness trackers a la FitBit. There's initial value from training yourself on what something looks/feels like … but diminishing returns after that. Whether there is more value to be found doesn't seem to matter. Factories would sensor up, go nuts with data, find one or two major insight, tire of data, and then just continue operating how they were before … but with a few…

[deleted]

Re: Big data is dead

#295

Post by vgt »

fuziontech wrote: post #282
ryadh wrote: post #277

While I get that they're sometimes useful to trigger debate, I don't really subscribe to very bold statements. We are drowning in data, it's all around us. Information overload is real. Data enables most of our daily digital experiences, from operational data to insights in the form of user facing analytics. Data systems are the backbone of the digital life. It's is an ocean and it's all about the vessel you pick to…

100% agree. One of the biggest assets we had at was the data we collected. We built models on it that would determine how markets were run and whether drivers and passengers were safe. These were key features that enabled us to bring a quality service to customers (over ye ol' taxi). The same applied to the autonomous cars, bikes, and scooters. We used data to improve placement of vehicles to help us anticipate and m…

Didn't know that Posthog is based on CH these days. Interesting!

Re: Big data is dead

#296

Post by disqard »

slt2021 wrote: post #36
fdgsdfogijq wrote: post #14

"For more than a decade now, the fact that people have a hard time gaining actionable insights from their data has been blamed on its size." The real issue is that business people usually ignore what the data says. Wading through data takes a huge amount of thought, which is in short supply. Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven". Most corpo…

I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick so…

An interesting essay that echoes these same sentiments:

https://ryxcommar.com/2022/11/27/goodbye-data-science/

Re: Big data is dead

#297

Post by gymbeaux »

fdgsdfogijq wrote: post #14

"For more than a decade now, the fact that people have a hard time gaining actionable insights from their data has been blamed on its size." The real issue is that business people usually ignore what the data says. Wading through data takes a huge amount of thought, which is in short supply. Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven". Most corpo…

Replace “data scientists” with “software engineers” and you have another accurate insight. They don’t want to listen to us about how to write software or derive value from data.

Re: Big data is dead

#298

Post by greggyb »

cmollis wrote: post #293

we regularly run audits on over 12 years of customer order histories. This requires scanning of about 40TB of data and growing. They used to jump through hoops on the Oracle cluster just to get data out for one customer. We pushed all of the order history into s3 parquet using Spark and I can query this in about 20 seconds using Spark or Presto. It's now streamed through kafka and Spark structured streaming so it's u…

I mean, you can almost fit all that data on one SSD. Micron's latest are 30TB. Commodity servers are available with 24 NVMe drive bays. At 7 GB/s read, across 24 drives, you can scan 40 TB in 240 seconds.

Such a server could readily fit 384 threads with dual EPYC and would be available with enough RAM to keep more than 10% of that data in cache.

Your workload absolutely fits the definition of "fits on one machine".

I am not suggesting here that you should put it on one machine. You definitely could, though.

Re: Big data is dead

#299

Post by gmfawcett »

ladyattis wrote: post #103
travisgriggs wrote: post #22

I've made anecdotal observations similiar to this over the last 10 years. I work in AgTech. A big push for a while here has been "more and more more data". Sensor-the-heck out of your farm, and We'll Tell You Things(tm). Most of what we as an industry are able to tell growers is stuff they already know or suspect. There is the occasional suprise or "Aha" moment where some correlation becomes apparent, but the thing a…

I think there's a problem at the heart of the matter, specifically the idea that the act of measurement is in itself powerful when in point of fact that this isn't universally the case. As the old adage goes: "garbage in, garbage out." Even more troubling, there is a physical limit to our ability to model what we measure. Take the retina, it has around a million light receptors and even if you assumed they only have…

> it has around a million light receptors and even if you assumed they only have two valid states then you're left with around 10^300,000 bits of information to process...

That would only be a million bits (1 Mb). You're counting potential states, not bits.

Re: Big data is dead

#300

Post by kulahan »

gffrd wrote: post #105
travisgriggs wrote: post #22

I've made anecdotal observations similiar to this over the last 10 years. I work in AgTech. A big push for a while here has been "more and more more data". Sensor-the-heck out of your farm, and We'll Tell You Things(tm). Most of what we as an industry are able to tell growers is stuff they already know or suspect. There is the occasional suprise or "Aha" moment where some correlation becomes apparent, but the thing a…

I've observed the same in manufacturing … and fitness trackers a la FitBit. There's initial value from training yourself on what something looks/feels like … but diminishing returns after that. Whether there is more value to be found doesn't seem to matter. Factories would sensor up, go nuts with data, find one or two major insight, tire of data, and then just continue operating how they were before … but with a few…

This is interesting, and I think you're on to something. I got a fitbit when I got serious about running because I had no idea what it felt like to run in zone 2. I can read about it, but that only gets you so far. While running I could actively check my heartrate and adjust. A year into the hobby and now I don't bring my fitbit on probably 50% of my runs. I know how fast I need to go.
Post reply on HN