Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead

motherduck.com

Post reply on HN
Print view View original

171–180 of 444 posts

Re: Big data is dead

#171

Post by hn_throwaway_99 »

guardiangod wrote: post #4

There is literally a post on front page on ChatGPT, and Microsoft and Google are preparing to duke it out starting in the _next 2 days_ over big-data generated 'chat' result. Big data was never going to be useful to even medium size enterprises, unless anyone can get public access to PBs of data, but that doesn't mean big data is dead. ChatGPT is literally changing how school will test their students, for a start. Ma…

> Maybe what the author is trying to say is 'small-scale big data is dead, but big data chugs on.'

That's pretty much exactly what the author says in the article.

Re: Big data is dead

#172

Post by threeseed »

0xB31B1B wrote: post #62
tomwheeler wrote: post #44

Earlier quoted context omitted.

Presumably the "order" you mention is a primary key to another table, likely one that references the individual items that make up that order, so the data will be much larger than you estimate. It will grow larger still if you include web logs from your e-commerce site and event data from your mobile app so that you can correlate these orders with items that customers considered but ultimately didn't buy. How will yo…

There are functionally less than 1000 organizations that currently require distributed compute for data analysis. You can get off the shelf AWS units with 1000 cores, terabytes of ram and storage, etc. The cost of compute has decreased faster than the amount of data we have to store and process. What we used to do with spark jobs we can do with python on a single box.

Let's assume your completely made-up 1000 organisations claim is true.

Right now I work for one of them: a global investment bank.

Within that organisation we have at least 100+ Spark clusters across the organisation doing distributed compute. And at least in our teams we have tight SLAs where a simple Python script simply can't deliver the results quick enough. Those jobs underpins 10s of billions of dollars in revenue and so for us money is not important, performance is.

So 1000 x 100 = 100,000 teams, all of whom I speak for, disagree with you.

Re: Big data is dead

#173

Post by mr_tristan »

spaintech wrote: post #54

Not that big data is dead, more like real time data is coming to life, but you need the old stuff around to make a buck or two… Well, that my view. LLMs are transformer model technique are making data more relevant than ever. If you are a business, well you are in for a “now real” digital transformation. Making data the centerpiece of your business business could mean that your effectiveness of business process could…

Yeah, I've noticed more applications just need to focus on making sense of raw information really quickly, but usually don't need an archive to make decisions.

There are lots of interesting things that can happen with "big streaming" than necessarily "big data". Like, cybersecurity is evolving to monitoring and reacting what everyone's machine is doing in the last 15 minutes, instead of having a huge database of hashes you trust. But not a ton of things really utilize what happened, say, 10 years ago on people's machines.

There's definitely some things that can use massive archives of old data, but I have found far, far fewer things that would benefit from it, and often that comes with some very big maintenance hassles. Most of the time, you can just set data retention to 30 days and be done.

Re: Big data is dead

#174

Post by maxfurman »

mordechai9000 wrote: post #110
edpichler wrote: post #91

I believe we are living in the "emotional era", so data has being ignored and 'feelings' come first when making decisions or creating processes. This is happening not only in companies but in our current society in general.

Perhaps I'm somewhat cynical, but I believe this is a feature of the human condition, not an attribute of our age in particular. Reason and analysis are tools that are used to justify what we already believe.

Agreed! The so-called "Age of Reason" was the anomaly, and probably not that much more reasonable than our own time.

Re: Big data is dead

#175

Post by CommieBobDole »

It's not dead, it's just entered the plateau of productivity. Where people use it for whatever it's useful for and don't try to solve every problem with it just because it's the cool new thing.

Re: Big data is dead

#176

Post by LarryMullins »

eganist wrote: post #29
humanizersequel wrote: post #23

Earlier quoted context omitted.

All math homework through the high school level is now as simple as figuring out how to describe it to ChatGPT (or maybe ChatGPT 2.0 for particularly tricky examples). Paper-writing is now a matter of figuring out how to rephrase LLM output in your own words to get around any watermarking or pattern detection.

Wolfram alpha has been around for math cheats (and people like me who just needed a more visual representation to learn) for a while now. Including proof of work.

Years before wolfram alpha, we had TI-89s with computer algebra systems for cheating your way through highschool math.

Re: Big data is dead

#177

Post by poorman »

This entire post reads like "you probably don't actually have big data".

What do these blockchains do that have to keep data around forever, with high throughput, and need to expose it quickly do? Are you saying they should delete parts of data in the chain?

Seriously, I've spent my career working on big data systems, and while the answer is sometimes "yes you need to delete your data", I don't think that's going to always work.

Re: Big data is dead

#178

Post by threeseed »

carlineng wrote: post #121
danielmarkbruce wrote: post #106

Earlier quoted context omitted.

Deliberately speculating so someone will correct it: I'd guess they'll make a bunch of enterprise tools to do things like: enable access and synch the data in a way which complies with various policy, encrypt/tokenize/hide certain columns etc, monitor queries, ensure data is encrypted at rest, stuff like that. Assuming the above it true: I'll bet the reason they aren't so loud about exactly what they are doing is the…

I'd wager you're right. All the "boring" stuff that's actually very complicated/difficult, and without which no large enterprise will adopt a technology.

Especially since enterprise companies hate the idea of shifting large amounts of highly sensitive company data onto commonly lost and misplaced work laptops.

If you're going to do that you better have your security and governance on point.

Re: Big data is dead

#179

Post by drowsspa »

nickersonm wrote: post #141
SoftTalker wrote: post #56

Earlier quoted context omitted.

Are we trying to produce adults who are able to think critically and creatively, and who reach their full intellectual potential, or are we trying to produce adults who can push a few buttons and blindly believe what the machine tells them?

> In fact, [writing] will introduce forgetfulness into the soul of those who learn it: they will not practice using their memory because they will put their trust in writing, which is external and depends on signs that belong to others, instead of trying to remember from the inside, completely on their own. - attributed to Socrates by Plato. c.399-347 BCE. “Phaedrus.”

Pretty sure there's a fallacy named after this whole "hey this is just exactly like before so we have nothing to be concerned about".

Re: Big data is dead

#180

Post by carabiner »

gffrd wrote: post #105
travisgriggs wrote: post #22

I've made anecdotal observations similiar to this over the last 10 years. I work in AgTech. A big push for a while here has been "more and more more data". Sensor-the-heck out of your farm, and We'll Tell You Things(tm). Most of what we as an industry are able to tell growers is stuff they already know or suspect. There is the occasional suprise or "Aha" moment where some correlation becomes apparent, but the thing a…

I've observed the same in manufacturing … and fitness trackers a la FitBit. There's initial value from training yourself on what something looks/feels like … but diminishing returns after that. Whether there is more value to be found doesn't seem to matter. Factories would sensor up, go nuts with data, find one or two major insight, tire of data, and then just continue operating how they were before … but with a few…

I used to track sooo much health and fitness data... Then I realized it mostly wasn't actionable, or at least, I wasn't altering my decisions based on it. The answer was always, "more training." So I stopped.
Post reply on HN