Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead

motherduck.com

Post reply on HN
Print view View original

211–220 of 444 posts

Re: Big data is dead

#211

Post by steveBK123 »

slt2021 wrote: post #36
fdgsdfogijq wrote: post #14

"For more than a decade now, the fact that people have a hard time gaining actionable insights from their data has been blamed on its size." The real issue is that business people usually ignore what the data says. Wading through data takes a huge amount of thought, which is in short supply. Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven". Most corpo…

I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick so…

Yes. I worked in the data org of a moderately sized financial firms tech org. The tech org claimed to be hugely data driven. Was in the org mottos and all of that.

Nonetheless, the CTO went on a multi-year, 10s of millions of dollars, huge data tech stack & staffing reorg shake up... with really zero data points explaining the driver, or what we would measure to determine it was successful.

So it became a self referential decision that we are successful by doing what he decided, and we are doing it because he decided it.

Re: Big data is dead

#212

Post by tootie »

guardiangod wrote: post #4

There is literally a post on front page on ChatGPT, and Microsoft and Google are preparing to duke it out starting in the _next 2 days_ over big-data generated 'chat' result. Big data was never going to be useful to even medium size enterprises, unless anyone can get public access to PBs of data, but that doesn't mean big data is dead. ChatGPT is literally changing how school will test their students, for a start. Ma…

That's a completely different topic. "Data" is obviously a pretty generic term and "large sets of data" are going to be more and more relevant to the world in general. What he's talking about is the Big Data trend in industry specifically around Business Intelligence (BI). That is, collecting as much data as possible on your users to optimize your product experience and profits. Tracking clicks, purchases, form dropoffs, email opens, ad impressions. It's mostly going to be first-party data (ie what did they do with our own products and content).

ChatGPT and the like are not going to get much use from that kind of data and instead are looking at a giant corpus of text and images scraped from a variety of public sources to infer what humans might think sounds smart. It's possible the two worlds will meet, but that's probably not what's going to be announced this week.

Re: Big data is dead

#214

Post by hugesniff »

"Very often when a data warehousing customer moves from an environment where they didn’t have separation of storage and compute into one where they do have it, their storage usage grows tremendously..."

Can someone explain why this is the case? Is it due to more replications or maintaining more indices?

Re: Big data is dead

#215

Post by tootie »

andix wrote: post #35

I see it all the time: people develop applications that will never ever get a database size of over 100GB and are using big data databases or distributed cloud databases. Often queries only hit a small subset of the date (one customer, one user). So you could easily fit everything into one SQL database. Using any of the traditional SQL databases takes away a lot of complications. You can do transactions, you can quer…

I think a lot of data tech has come full circle is now mostly just relational databases. Our org is invested in redshift which lets us mostly pay as we go. The DB itself is just a Postgres facade on scalable storage with some native connectors to file stores and third-parties. After rolling over our stack like three times, we're now just dumping tons of raw data into staging tables, then creating views on top of them. It's 97% raw SQL with a smattering of python for clunky extractions. And we're now true believers in ELT vs ETL.

Re: Big data is dead

#216

Post by spaintech »

threeseed wrote: post #194
spaintech wrote: post #54

Not that big data is dead, more like real time data is coming to life, but you need the old stuff around to make a buck or two… Well, that my view. LLMs are transformer model technique are making data more relevant than ever. If you are a business, well you are in for a “now real” digital transformation. Making data the centerpiece of your business business could mean that your effectiveness of business process could…

I assume you've never actually worked at a bank. They've been working to implement your ideas for decades and none of it requires LLMs or any machine learning techniques. Basic old ETL is more than sufficient. The issue is that (a) the calculations they need to perform are complex and take time to run (b) there are financial regulations that weave its way through those system and (c) there is a lot of legacy code esp…

Well, that is an understatement. I do agree with you that banks have been trying to fix decades old application.

But in this process, you don’t need ETL, nor all the process and development to accomplish these ideas. Conceptually the idea builds its self (it learns) how to threat the data, quite revealing and near real time. Considering you account for security and privacy, then you basically shift your input into the data stream and using a natural language get the data output you need, not clunky apps.

Imagine I just login, and say: me>how much do I have? bank>You have 100$ me>Please send 50$ to 1003 bank> Are you sure? Please add your security code to confirm

bla bla…

All this with little intervention.

Banks spend hundreds of man hours developing a lacking application while delivering a very poor customer experience. They spend millions on running decades old applications because it so expensive to change them… and thus the circle continues…

I’m really exited to see DataBases disappear conceptualy, data entry, mostly all that just disappear… I will ask my ChapBot for statement, give me a personal investment advice, and classify all my purchases and see where my wife has been spending all my money, all from the confort of my phone.

it’s a brave new wold we are wakening up to, that to me is exciting. And coming from having helped several major banks build their infrastructure, it’s just a boost to talk about something fresh, no more Hypervisor, core count, db licenses, ect. Ok, I’ll concede it’s pretty much the same old, just the nemonics will be different… How many GPUs, how quickly can you spin a container, how fast if your S3 datastore… oh wait, there is that circle again… >:D

Re: Big data is dead

#217

Post by angry_moose »

nerpderp82 wrote: post #153

Big Data was whatever someone couldn't handle in a spreadsheet or on their laptop using R. This paper is 8 years old and it was somewhat obvious then. Scalability! But at what COST? https://www.usenix.org/system/files/conference/hotos15/hotos... A big single machine can handle 98% of peoples data reduction needs. This has always been true. Just because your laptop only has 16GB doesn't mean you need a Hadoop (or Spar…

My experience with "Big Data" is it was something that couldn't be handled in a spreadsheet or on their laptop using R because it was so inefficiently coded.

I got sucked into "weekly key metric takes over 14 hours to run on our multi-node kubernetes cluster" a while back. I'm not sure how many nodes it actually used, nor did I really care.

Digging into it, the python code ingested about ~50GB of various files, made well over a dozen copies of everything, leaving the whole thing extremely memory starved. I replaced almost all of the program with some "grep | sed | awk | sed | grep" abomination that stripped about 98% of the unnecessary info first and it ran in under 2 minutes on my laptop. I probably should have tightened it up more but I was more than happy to wash my hands of the whole thing by that point.

Instead of improving the code, they just kept tossing more compute at it. Still heard all kinds of grumbling about os.system('grep | sed | awk | sed | grep') not being "pythonic" and "bad practice"; but not enough that they actually bothered to fix it.

Re: Big data is dead

#218

Post by tootie »

edpichler wrote: post #91

I believe we are living in the "emotional era", so data has being ignored and 'feelings' come first when making decisions or creating processes. This is happening not only in companies but in our current society in general.

I think there's absolutely a place for this. I often of the old Henry Ford quote about people wanting faster horses. Data and analytics are great for optimization, but sometimes you need to trust your gut and give people something they didn't ask for to have a breakthrough.

Re: Big data is dead

#219

Post by nerpderp82 »

angry_moose wrote: post #217
nerpderp82 wrote: post #153

Big Data was whatever someone couldn't handle in a spreadsheet or on their laptop using R. This paper is 8 years old and it was somewhat obvious then. Scalability! But at what COST? https://www.usenix.org/system/files/conference/hotos15/hotos... A big single machine can handle 98% of peoples data reduction needs. This has always been true. Just because your laptop only has 16GB doesn't mean you need a Hadoop (or Spar…

My experience with "Big Data" is it was something that couldn't be handled in a spreadsheet or on their laptop using R because it was so inefficiently coded. I got sucked into "weekly key metric takes over 14 hours to run on our multi-node kubernetes cluster" a while back. I'm not sure how many nodes it actually used, nor did I really care. Digging into it, the python code ingested about ~50GB of various files, made…

That is one of the selling points of Hadoop, you can write garbage code and scale your way out of any problem, turning the $$$ knob up to more nodes.

Re: Big data is dead

#220

Post by didgetmaster »

pjmorris wrote: post #90
slt2021 wrote: post #36

Earlier quoted context omitted.

I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick so…

> I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. So, a synonym for 'consultant?' :)

In the news business, if your story or opinion backs up the preconceived notions of the investigative reporter then you are a 'source' otherwise you are a 'conspiracy theorist'.
Post reply on HN