Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead

motherduck.com

Post reply on HN
Print view View original

141–150 of 444 posts

Re: Big data is dead

#141

Post by nickersonm »

SoftTalker wrote: post #56
criddell wrote: post #41

Earlier quoted context omitted.

Is it crazy to think that instead of stepping up in the war against AI we instead try to figure out a way to teach kids assuming they will use AIs?

Are we trying to produce adults who are able to think critically and creatively, and who reach their full intellectual potential, or are we trying to produce adults who can push a few buttons and blindly believe what the machine tells them?

> In fact, [writing] will introduce forgetfulness into the soul of those who learn it: they will not practice using their memory because they will put their trust in writing, which is external and depends on signs that belong to others, instead of trying to remember from the inside, completely on their own.

- attributed to Socrates by Plato. c.399-347 BCE. “Phaedrus.”

Re: Big data is dead

#142

Post by carlineng »

vgt wrote: post #124
carlineng wrote: post #121

Earlier quoted context omitted.

I'd wager you're right. All the "boring" stuff that's actually very complicated/difficult, and without which no large enterprise will adopt a technology.

Shoot me a note and I'm happy to fill you in! (PM at MotherDuck)

Would love to! How do I get in touch? My contact info is in my profile.

Re: Big data is dead

#143

Post by koolba »

slt2021 wrote: post #36
fdgsdfogijq wrote: post #14

"For more than a decade now, the fact that people have a hard time gaining actionable insights from their data has been blamed on its size." The real issue is that business people usually ignore what the data says. Wading through data takes a huge amount of thought, which is in short supply. Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven". Most corpo…

I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick so…

> if you want to succeed as a Data Scientist and be praised by management - you got to provide data analysis that supports managements ideas (however wrong or ineffective they might be).

> Data Scientist's job is to launder management's intuition using quantitative methods :)

It’s no different than the days when grey bearded wisemen would read the stars and weave a tale about the great glory that awaits the king if he proceeds with whatever he already wants to do.

The beards might be a bit shorter or nonexistent, but the story hasn’t changed.

Re: Big data is dead

#144

Post by throwmeout123 »

airstrike wrote: post #86
guardiangod wrote: post #4

There is literally a post on front page on ChatGPT, and Microsoft and Google are preparing to duke it out starting in the _next 2 days_ over big-data generated 'chat' result. Big data was never going to be useful to even medium size enterprises, unless anyone can get public access to PBs of data, but that doesn't mean big data is dead. ChatGPT is literally changing how school will test their students, for a start. Ma…

> ChatGPT is literally changing how school will test their students, for a start. Here's a novel idea: test students using pen and paper?

In Italy we test with pen and paper and the quality of italian schools is abysmal. The point is not the test, is the quality of the teachers. They are the only hope to form good humans, not some standardized test.

Re: Big data is dead

#145

Post by CobrastanJorji »

wizwit999 wrote: post #136

Perhaps this is true for business data (though I'm skeptical of the claims), but, for example, for security data, this isn't true at all. Collecting cloud, identity, SaaS, and network logs/data can easily exceed hundreds of terabytes. A big reason why we're building Matano as a data lake for security. It seems an odd pitch in general to say, hey my product specifically performs poorly on large datasets.

On the contrary, identifying what your product is explicitly not aiming to do is extremely helpful. "Big" adds a lot of complexity and pain, most people don't do that, our product avoids the complexity and pain and is the best choice for most people. Seems like a good, simple pitch, and all it requires is the humility to say that your solution isn't the best for some use cases.

Re: Big data is dead

#146

Post by 0xdeadbeefbabe »

travisgriggs wrote: post #22

I've made anecdotal observations similiar to this over the last 10 years. I work in AgTech. A big push for a while here has been "more and more more data". Sensor-the-heck out of your farm, and We'll Tell You Things(tm). Most of what we as an industry are able to tell growers is stuff they already know or suspect. There is the occasional suprise or "Aha" moment where some correlation becomes apparent, but the thing a…

> goes geek-crazy for the first year or so

The problem is that they don't stay geek-crazy?

Re: Big data is dead

#147

Post by remus »

slt2021 wrote: post #36
fdgsdfogijq wrote: post #14

"For more than a decade now, the fact that people have a hard time gaining actionable insights from their data has been blamed on its size." The real issue is that business people usually ignore what the data says. Wading through data takes a huge amount of thought, which is in short supply. Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven". Most corpo…

I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick so…

I think this depends a lot on the org. In a place I used to work we collected and analysed a lot of data which convinced management to significantly change the spec of the product and spend a lot more time and effort on testing, because the product was being used in unexpected ways.

I would say it was a very engineering driven org however, so if you could present compelling data it could go a long way.

Re: Big data is dead

#148

Post by danielmarkbruce »

vgt wrote: post #124
carlineng wrote: post #121

Earlier quoted context omitted.

I'd wager you're right. All the "boring" stuff that's actually very complicated/difficult, and without which no large enterprise will adopt a technology.

Shoot me a note and I'm happy to fill you in! (PM at MotherDuck)

Is there a reason you can't post it here?

Re: Big data is dead

#149

Post by politician »

0cf8612b2e1e wrote: post #112
airstrike wrote: post #86

Earlier quoted context omitted.

> ChatGPT is literally changing how school will test their students, for a start. Here's a novel idea: test students using pen and paper?

Teachers assign better scores to papers with better penmanship. I forget how strong the effect was, but using a keyboard does help equalize some biases.

Scan the hand-written work back into a digital format and present the results to the teachers for evaluation in their preferred typeface.

Re: Big data is dead

#150

Post by Loughla »

stcroixx wrote: post #108
Loughla wrote: post #30

Earlier quoted context omitted.

I mean, that's certainly part of it. But you're also seeing very fast restructuring of individual courses (because programs overall will definitely take years to restructure because higher education moves so damned slow) to account for these tools. In the small institution I am currently working with, the English courses, in one week, integrated chatgpt as a tool for students to work with. It's part of the collaborat…

Sounds like they simply panicked and threw something together with little thought or preparation, what, right in the middle of an actual course? And they want to charge kids for this kind of 'expert instruction'? I'd be pissed as a student.

What a weird assumption you made there. In what way does what I wrote sound panicked? Because it was a week? Yes it was fast, but it was a massive effort of the entire english faculty.

It's integrated into existing assignments, modifying processes that are super well established already. It was like integrating a new person into the class. Also, it was before semester, so the students literally saw nothing weird; that was another strange assumption for you to make.

Just a tip, and don't read tone in this statement, but don't assume things. 9/10 times you're going to be incorrect. It's much better to ask questions, instead of making statements with question marks at the end of them.

Post reply on HN