Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead

motherduck.com

Post reply on HN
Print view View original

191–200 of 444 posts

Re: Big data is dead

#191

Post by maliker »

miguelazo wrote: post #11
guardiangod wrote: post #4

There is literally a post on front page on ChatGPT, and Microsoft and Google are preparing to duke it out starting in the _next 2 days_ over big-data generated 'chat' result. Big data was never going to be useful to even medium size enterprises, unless anyone can get public access to PBs of data, but that doesn't mean big data is dead. ChatGPT is literally changing how school will test their students, for a start. Ma…

>ChatGPT is literally changing how school will test their students, for a start. Sure, instead of schools checking for plagiarism from other students' papers using turnitin.com, they'll check for plagiarism using ChatGPT tools that scan for known output from their industrial-scale amalgamation of plagiarized materials. Big whoop.

It appears that it's hard to detect AI generated content. E.g. true detection rates are only around 25% and there are also techniques to further mask output [1].

[1] https://www.nbcnews.com/tech/innovation/chatgpt-can-help-foo...

Re: Big data is dead

#192

Post by ilyt »

ravivooda wrote: post #163
danuker wrote: post #6

I agree with many of the points here. My cheap no-name old laptop SSD writes with 170MB/s. A customer has a name, address, email and order. Let's say 200 bytes for each. That means I can write 844000 new customers per second, far outside my personal marketing reach. My disk is 240GB, which means I can store data for 1.2 billion customers. It'll take a while until I become that successful.

It never sat well with me that none of the production services could leverage my local computation and storage power. I don't need to store my contacts on a remote server that could index my contacts when mixed with every other contact in a single table. That’s a blatantly oversimplified example but you get the gist.

Developing apps as local-mostly with remote being "just storage" might've been interesting approach but oh so many stuff moved to webshit from native apps and browsers still don't even have decent data management.

Re: Big data is dead

#193

Post by commandlinefan »

fdgsdfogijq wrote: post #14

"For more than a decade now, the fact that people have a hard time gaining actionable insights from their data has been blamed on its size." The real issue is that business people usually ignore what the data says. Wading through data takes a huge amount of thought, which is in short supply. Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven". Most corpo…

> a huge amount of thought

Which itself both takes time and is wildly unpredictable - neither of play well with today's Taylorist managements schemes.

Re: Big data is dead

#194

Post by threeseed »

spaintech wrote: post #54

Not that big data is dead, more like real time data is coming to life, but you need the old stuff around to make a buck or two… Well, that my view. LLMs are transformer model technique are making data more relevant than ever. If you are a business, well you are in for a “now real” digital transformation. Making data the centerpiece of your business business could mean that your effectiveness of business process could…

I assume you've never actually worked at a bank.

They've been working to implement your ideas for decades and none of it requires LLMs or any machine learning techniques. Basic old ETL is more than sufficient.

The issue is that (a) the calculations they need to perform are complex and take time to run (b) there are financial regulations that weave its way through those system and (c) there is a lot of legacy code especially in the core ledger system which "just works" and people are reluctant to touch.

That said depending on your bank you can get real-time account activity, loan approvals in < 5 minutes etc.

Re: Big data is dead

#195

Post by hef19898 »

tracerbulletx wrote: post #167
fdgsdfogijq wrote: post #14

"For more than a decade now, the fact that people have a hard time gaining actionable insights from their data has been blamed on its size." The real issue is that business people usually ignore what the data says. Wading through data takes a huge amount of thought, which is in short supply. Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven". Most corpo…

I find a lot of organizations don't have the discipline to harness whatever power their data may have. Sure collect everything, but god forbid you have any sort of data governance, or spend a single resource minute of time manually tagging or organizing or validating it. Then they try to make shitty ML models or products out of it, but don't care if the models actually work or not, just that they have AI now. Then a…

Palantir, you have to have Palantir.

Oh, and a bunch of data scientists with zero domain knowledge for whatever data they are analyzing, preferably with PhDs in maths, but some ML background will do. And agile, because of course all those Palantir dashboards can only be developed using agile.

Once all is said and done, zero insight was created but a whole lot of consultants, contractors and project managers have been paid handsomely, while some higher ups can now put "implemented agile and big data at X" on their LinkedIn profiles.

Re: Big data is dead

#196

Post by sva_ »

slt2021 wrote: post #36
fdgsdfogijq wrote: post #14

"For more than a decade now, the fact that people have a hard time gaining actionable insights from their data has been blamed on its size." The real issue is that business people usually ignore what the data says. Wading through data takes a huge amount of thought, which is in short supply. Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven". Most corpo…

I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick so…

> I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs.

Also someone to blame if it doesn't work out

Re: Big data is dead

#197

Post by pcthrowaway »

Loughla wrote: post #150
stcroixx wrote: post #108

Earlier quoted context omitted.

Sounds like they simply panicked and threw something together with little thought or preparation, what, right in the middle of an actual course? And they want to charge kids for this kind of 'expert instruction'? I'd be pissed as a student.

What a weird assumption you made there. In what way does what I wrote sound panicked? Because it was a week? Yes it was fast, but it was a massive effort of the entire english faculty. It's integrated into existing assignments, modifying processes that are super well established already. It was like integrating a new person into the class. Also, it was before semester, so the students literally saw nothing weird; tha…

> don't assume things. 9/10 times you're going to be incorrect

Isn't that... an assumption?

Re: Big data is dead

#198

Post by alar44 »

slt2021 wrote: post #36
fdgsdfogijq wrote: post #14

"For more than a decade now, the fact that people have a hard time gaining actionable insights from their data has been blamed on its size." The real issue is that business people usually ignore what the data says. Wading through data takes a huge amount of thought, which is in short supply. Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven". Most corpo…

I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick so…

I mean, that makes sense does it not? If you're confirming something people already had a hunch about, why would they challenge it? And if it does go against their belief, they are going to want to make sure the data is correct before they change the course of the ship.

Re: Big data is dead

#199

Post by nerpderp82 »

mywittyname wrote: post #162
nerpderp82 wrote: post #153

Big Data was whatever someone couldn't handle in a spreadsheet or on their laptop using R. This paper is 8 years old and it was somewhat obvious then. Scalability! But at what COST? https://www.usenix.org/system/files/conference/hotos15/hotos... A big single machine can handle 98% of peoples data reduction needs. This has always been true. Just because your laptop only has 16GB doesn't mean you need a Hadoop (or Spar…

You can do a petabytes of analysis with regular old BigQuery just as easily as you can analyze megabytes of data. This solves the scalability issue for a lot of companies, IMHO.

I agree, BQ is a gem on GCP. You pay for storage (or not, you can use federated queries) and don't pay anything when you aren't using it. The ability to dynamically scale reservations is pretty nice as well.

Re: Big data is dead

#200

Post by eganist »

LarryMullins wrote: post #176
eganist wrote: post #29

Earlier quoted context omitted.

Wolfram alpha has been around for math cheats (and people like me who just needed a more visual representation to learn) for a while now. Including proof of work.

Years before wolfram alpha, we had TI-89s with computer algebra systems for cheating your way through highschool math.

Oh yeah, it's why most of my classes restricted us to TI-83s. The TI-89 was restricted in schools to basically calc and above, and the TI-92 was just banned. Lol
Post reply on HN