Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead

motherduck.com

Post reply on HN
Print view View original

101–110 of 444 posts

Re: Big data is dead

#101

Post by strbean »

pjmorris wrote: post #90
slt2021 wrote: post #36

Earlier quoted context omitted.

I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick so…

> I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. So, a synonym for 'consultant?' :)

If Data Scientists are essentially in-house management consultants, I wonder which is cheaper?

Re: Big data is dead

#102

Post by htrp »

pjmorris wrote: post #90
slt2021 wrote: post #36

Earlier quoted context omitted.

I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick so…

> I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. So, a synonym for 'consultant?' :)

All symptoms of the same problem..... you can hire McKinsey to confirm your priors, massage the data to confirm your priors, or anything in between.

Re: Big data is dead

#103

Post by ladyattis »

travisgriggs wrote: post #22

I've made anecdotal observations similiar to this over the last 10 years. I work in AgTech. A big push for a while here has been "more and more more data". Sensor-the-heck out of your farm, and We'll Tell You Things(tm). Most of what we as an industry are able to tell growers is stuff they already know or suspect. There is the occasional suprise or "Aha" moment where some correlation becomes apparent, but the thing a…

I think there's a problem at the heart of the matter, specifically the idea that the act of measurement is in itself powerful when in point of fact that this isn't universally the case. As the old adage goes: "garbage in, garbage out." Even more troubling, there is a physical limit to our ability to model what we measure. Take the retina, it has around a million light receptors and even if you assumed they only have two valid states then you're left with around 10^300,000 bits of information to process, so good luck with that. Same thing applies to whatever firms are measuring and what they think is conveying relevant information as they'll have similarly exponential increases if they don't filter out the vast majority of irrelevant data points and states.

Re: Big data is dead

#104

Post by snarf21 »

guardiangod wrote: post #4

There is literally a post on front page on ChatGPT, and Microsoft and Google are preparing to duke it out starting in the _next 2 days_ over big-data generated 'chat' result. Big data was never going to be useful to even medium size enterprises, unless anyone can get public access to PBs of data, but that doesn't mean big data is dead. ChatGPT is literally changing how school will test their students, for a start. Ma…

Big Data drives the most profitable and society bending changes of all time, just to serve us better Ads.

Re: Big data is dead

#105

Post by gffrd »

travisgriggs wrote: post #22

I've made anecdotal observations similiar to this over the last 10 years. I work in AgTech. A big push for a while here has been "more and more more data". Sensor-the-heck out of your farm, and We'll Tell You Things(tm). Most of what we as an industry are able to tell growers is stuff they already know or suspect. There is the occasional suprise or "Aha" moment where some correlation becomes apparent, but the thing a…

I've observed the same in manufacturing … and fitness trackers a la FitBit.

There's initial value from training yourself on what something looks/feels like … but diminishing returns after that. Whether there is more value to be found doesn't seem to matter.

Factories would sensor up, go nuts with data, find one or two major insight, tire of data, and then just continue operating how they were before … but with a few new operational tools in their quiver.

Same is true of fitness trackers: you excitedly get one, learn how much you really are sitting(!), adjust your patterns, time passes … then one day you realize you haven't put it on for a week. It stays in the drawer.

Not unless they're threatened with ruin will people make changes to the standard way of doing things. This is actually … not bad! Continuity is important, and this is kind of a subconscious gating function to prevent deviation from a proven way of working. So, the change has to be so compelling or so pressing that they're forced to. Not a bad thing.

While we think things change overnight in this world, they generally take awhile … stay patient … it's worth it.

Re: Big data is dead

#106

Post by danielmarkbruce »

carlineng wrote: post #84

MotherDuck has been making the rounds with a big funding announcement [1], and a lot of posts like this one. As a life-long data industry person, I agree with nearly all of what Jordan and Ryan are saying. It all tracks with my personal experience on both the customer and vendor side of "Big Data". That being said, what's the product? The website says "Commercializing DuckDB", but that doesn't give much of an idea of…

Deliberately speculating so someone will correct it: I'd guess they'll make a bunch of enterprise tools to do things like: enable access and synch the data in a way which complies with various policy, encrypt/tokenize/hide certain columns etc, monitor queries, ensure data is encrypted at rest, stuff like that.

Assuming the above it true: I'll bet the reason they aren't so loud about exactly what they are doing is they want to get a head start on it. In theory anyone can build this stuff around DuckDB. From a marketing perspective the clever thing to do would be drive up usage of DuckDB while they build out all this functionality and then the minute corporates start seeing problems with their people using it (compliance etc), they have the solutions.

Re: Big data is dead

#108

Post by stcroixx »

Loughla wrote: post #30
miguelazo wrote: post #11

Earlier quoted context omitted.

>ChatGPT is literally changing how school will test their students, for a start. Sure, instead of schools checking for plagiarism from other students' papers using turnitin.com, they'll check for plagiarism using ChatGPT tools that scan for known output from their industrial-scale amalgamation of plagiarized materials. Big whoop.

I mean, that's certainly part of it. But you're also seeing very fast restructuring of individual courses (because programs overall will definitely take years to restructure because higher education moves so damned slow) to account for these tools. In the small institution I am currently working with, the English courses, in one week, integrated chatgpt as a tool for students to work with. It's part of the collaborat…

Sounds like they simply panicked and threw something together with little thought or preparation, what, right in the middle of an actual course? And they want to charge kids for this kind of 'expert instruction'? I'd be pissed as a student.

Re: Big data is dead

#109

Post by slt2021 »

strbean wrote: post #101
pjmorris wrote: post #90

Earlier quoted context omitted.

> I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. So, a synonym for 'consultant?' :)

If Data Scientists are essentially in-house management consultants, I wonder which is cheaper?

This could be a reason why Data Scientist as a job title exploded in last years, every middle manager could afford one/two/few headcounts of data scientists to produce analysis that advances that middle manager's corporate agenda (more growth, empire building, expansion to certain de-novo areas, etc).

Recent tech layoffs is the other side of that growth, when cheap money is gone and company is forced to stick to core competencies and shutdown growth plans

Re: Big data is dead

#110

Post by mordechai9000 »

edpichler wrote: post #91

I believe we are living in the "emotional era", so data has being ignored and 'feelings' come first when making decisions or creating processes. This is happening not only in companies but in our current society in general.

Perhaps I'm somewhat cynical, but I believe this is a feature of the human condition, not an attribute of our age in particular. Reason and analysis are tools that are used to justify what we already believe.
Post reply on HN