Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead

motherduck.com

Post reply on HN
Print view View original

121–130 of 444 posts

Re: Big data is dead

#121

Post by carlineng »

danielmarkbruce wrote: post #106
carlineng wrote: post #84

MotherDuck has been making the rounds with a big funding announcement [1], and a lot of posts like this one. As a life-long data industry person, I agree with nearly all of what Jordan and Ryan are saying. It all tracks with my personal experience on both the customer and vendor side of "Big Data". That being said, what's the product? The website says "Commercializing DuckDB", but that doesn't give much of an idea of…

Deliberately speculating so someone will correct it: I'd guess they'll make a bunch of enterprise tools to do things like: enable access and synch the data in a way which complies with various policy, encrypt/tokenize/hide certain columns etc, monitor queries, ensure data is encrypted at rest, stuff like that. Assuming the above it true: I'll bet the reason they aren't so loud about exactly what they are doing is the…

I'd wager you're right. All the "boring" stuff that's actually very complicated/difficult, and without which no large enterprise will adopt a technology.

Re: Big data is dead

#122

Post by e12e »

travisgriggs wrote: post #22

I've made anecdotal observations similiar to this over the last 10 years. I work in AgTech. A big push for a while here has been "more and more more data". Sensor-the-heck out of your farm, and We'll Tell You Things(tm). Most of what we as an industry are able to tell growers is stuff they already know or suspect. There is the occasional suprise or "Aha" moment where some correlation becomes apparent, but the thing a…

But isn't this the essence of industrialization and automation? Messure, adjust process; repeat until feedback loop is stable - document and keep doing the thing that works, over and over?

If you want Toyota style continuous improvement you would need to improve in new areas of the process / new metrics, most of the time?

Re: Big data is dead

#123

Post by ArjenM »

eska wrote: post #66
slt2021 wrote: post #36

Earlier quoted context omitted.

I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick so…

This would match what psychologists say about humans in general: we feel first, then we use our brain to justify that feeling. We’re not rational beings.

A whole industry of emotional branding is thriving, systematically overloading our brains so it hurt to even think differently in the moment.

We are accepting a whole lot of assumptions every day.

Re: Big data is dead

#124

Post by vgt »

carlineng wrote: post #121
danielmarkbruce wrote: post #106

Earlier quoted context omitted.

Deliberately speculating so someone will correct it: I'd guess they'll make a bunch of enterprise tools to do things like: enable access and synch the data in a way which complies with various policy, encrypt/tokenize/hide certain columns etc, monitor queries, ensure data is encrypted at rest, stuff like that. Assuming the above it true: I'll bet the reason they aren't so loud about exactly what they are doing is the…

I'd wager you're right. All the "boring" stuff that's actually very complicated/difficult, and without which no large enterprise will adopt a technology.

Shoot me a note and I'm happy to fill you in!

(PM at MotherDuck)

Re: Big data is dead

#125

Post by esel2k »

travisgriggs wrote: post #22

I've made anecdotal observations similiar to this over the last 10 years. I work in AgTech. A big push for a while here has been "more and more more data". Sensor-the-heck out of your farm, and We'll Tell You Things(tm). Most of what we as an industry are able to tell growers is stuff they already know or suspect. There is the occasional suprise or "Aha" moment where some correlation becomes apparent, but the thing a…

Very interesting. Left AgTech last year but had similar experiences, even worse where often the single most prominent use-case was to follow some painful necessary documentation of ag inputs (chemical, seeds, fertiliser) to get subsidies. Real inputs from data? Nah!

Re: Big data is dead

#126

Post by MonkeyMalarky »

0cf8612b2e1e wrote: post #112
airstrike wrote: post #86

Earlier quoted context omitted.

> ChatGPT is literally changing how school will test their students, for a start. Here's a novel idea: test students using pen and paper?

Teachers assign better scores to papers with better penmanship. I forget how strong the effect was, but using a keyboard does help equalize some biases.

Typewriters for everyone!

Re: Big data is dead

#128

Post by lostmsu »

danuker wrote: post #6

I agree with many of the points here. My cheap no-name old laptop SSD writes with 170MB/s. A customer has a name, address, email and order. Let's say 200 bytes for each. That means I can write 844000 new customers per second, far outside my personal marketing reach. My disk is 240GB, which means I can store data for 1.2 billion customers. It'll take a while until I become that successful.

> 170MB/s

That is not random access speed. For random access my relatively high-performance SSD only does 42MB/s reading and 80MB/s writing.

Re: Big data is dead

#129

Post by albert_e »

0cf8612b2e1e wrote: post #112
airstrike wrote: post #86

Earlier quoted context omitted.

> ChatGPT is literally changing how school will test their students, for a start. Here's a novel idea: test students using pen and paper?

Teachers assign better scores to papers with better penmanship. I forget how strong the effect was, but using a keyboard does help equalize some biases.

Keyboards and screens in an exam hall then?

Re: Big data is dead

#130

Post by H8crilA »

Big data starts somewhere around a petabyte, maybe a bit lower than that. That's when you need some serious, dedicated systems. But as always everyone wants to (pretend to) do what the big players do.
Post reply on HN