Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead

motherduck.com

Post reply on HN
Print view View original

241–250 of 444 posts

Re: Big data is dead

#241

Post by LeanderK »

Who has ever believed those claims? There's a common saying "garbage in, garbage out" about what happens with all those fancy models if the data quality is not high. That's really independent from dataset-size. There's no magic insight you get because your dataset is bigger. You need a quality analyst to handle your data, irrelevant of its size.

Also, who thought their company would cease to function because surely they will hit google-scale dataset-sizes in the near future? Impossible for most except the biggest of the biggest

Re: Big data is dead

#242

Post by boh »

fdgsdfogijq wrote: post #14

"For more than a decade now, the fact that people have a hard time gaining actionable insights from their data has been blamed on its size." The real issue is that business people usually ignore what the data says. Wading through data takes a huge amount of thought, which is in short supply. Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven". Most corpo…

The other thing to consider is that data simply has nothing of value. Part of the marketing of big data is the almost fairy tale belief in "insights" existing in any data set if you just look hard enough.

Re: Big data is dead

#243

Post by neuronic »

happymellon wrote: post #96
pjmorris wrote: post #90

Earlier quoted context omitted.

> I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. So, a synonym for 'consultant?' :)

My experience with consultants normally ended up with them asking why they are there and what report should they present to upper management. I've always used them as "independent 3rd parties" who were listened to.

As a consultant with roots in backend dev, I fully understand the scrutiny that we receive because unfortunately, it is often very warranted... It feels a bit refreshing to read your comment and see someone articulate what I am trying to convey to my clients. I am a tool, and yes, this pun is intended.

Re: Big data is dead

#244

Post by zX41ZdbW »

My presentation from FOSDEM 2023 is very sympathetic to the "Big data is dead" statement: https://www.youtube.com/watch?v=JlcI2Vfz_uk

It is about using modern tools (ClickHouse) for data engineering without the fluff - when you can take whatever dataset or data stream and make what you need without the need for complex infrastructure.

Nevertheless, the statement "big data is dead" is short-sighted, and I don't entirely follow this opinion.

For example, here is one of ClickHouse's use-case:

> Main cluster is 110PB nvme storage, 100k+ cpu cores, 800TB ram. The uncompressed data size on the main cluster is 1EB.

And when you have this sort of data for realtime processing, no other technology can help you.

Re: Big data is dead

#245

Post by ProjectArcturis »

m463 wrote: post #99
slt2021 wrote: post #36

Earlier quoted context omitted.

I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick so…

I wonder what a data scientist could really find out about executive (over?) compensation. employee compensation. working from home. office cubicle size and layout. tool expenditure for employees vs productivity.

How would you measure productivity at scale?

Re: Big data is dead

#246

Post by hinkley »

danuker wrote: post #6

I agree with many of the points here. My cheap no-name old laptop SSD writes with 170MB/s. A customer has a name, address, email and order. Let's say 200 bytes for each. That means I can write 844000 new customers per second, far outside my personal marketing reach. My disk is 240GB, which means I can store data for 1.2 billion customers. It'll take a while until I become that successful.

One of my "computers are really fucking fast" experiments, almost a decade ago, was when I was trying to do a histogram plot of a function that I was 98% sure was terribly broken. It was expected to give a uniform distribution so I figured I'll just plot a bunch of values into a 2d space and then convert it to a greyscale image.

At first I tried to puzzle out a good sampling strategy to make sure I didn't bias the output, then on a whim I tried 2^32 samples and went to lunch. It took something like a half an hour to do 4 billion samples. Took me a couple times to figure out how to squeeze 4k megapixels into a graph so I ran it a few more times, but the results showed a very distinct banding pattern that confirmed that the problem was every bit as bad as I suspected, which was a blocking issue for our release. A couple of hours well spent, running through an 'intractable problem' that really wasn't.

Re: Big data is dead

#247

Post by taftster »

Mizza wrote: post #189
airstrike wrote: post #86

Earlier quoted context omitted.

> ChatGPT is literally changing how school will test their students, for a start. Here's a novel idea: test students using pen and paper?

Or, preferably, admit that testing wasn't a good idea to begin with and focus on optimizing children for learning, not test-taking.

I don't even think we're just talking about children either. Test taking in academia (university and above) could stand a much needed fresh look.

I am hopeful that a change happens in academia to prepare students for jobs, which is why they are going to school in the first place. Yes, students need to learn how to "think", but really they are wanting to get the technical skills to perform their duties more than anything.

We have bestowed too much credence in traditional academia not useful to the average person or average job. College is a "game" for most students, and they put up going through the motions of testing, etc. for the sake of the diploma at the end.

I hope we're going to enter a new era of what college means for those looking to get something different out of it.

Re: Big data is dead

#248

Post by threeseed »

fho wrote: post #139
doug_durham wrote: post #85

Earlier quoted context omitted.

Citations please? That's a pretty bold statement to make in the face of observed reality.

https://yourdatafitsinram.net/

This is such a lazy response.

I/O performance is just one of many characteristics that impact performance and from experience the one you least need to worry about. RAID 0 across multiple high-end NVME drives with OS file caching is going to be more than fast enough for most use cases.

The issue is running out of CPU performance and being able to seamlessly scale up/down compute with live running workloads.

Re: Big data is dead

#250

Post by threeseed »

tootie wrote: post #215
andix wrote: post #35

I see it all the time: people develop applications that will never ever get a database size of over 100GB and are using big data databases or distributed cloud databases. Often queries only hit a small subset of the date (one customer, one user). So you could easily fit everything into one SQL database. Using any of the traditional SQL databases takes away a lot of complications. You can do transactions, you can quer…

I think a lot of data tech has come full circle is now mostly just relational databases. Our org is invested in redshift which lets us mostly pay as we go. The DB itself is just a Postgres facade on scalable storage with some native connectors to file stores and third-parties. After rolling over our stack like three times, we're now just dumping tons of raw data into staging tables, then creating views on top of them…

Redshift with S3 storage is no different to Spark SQL with S3 storage.

Both are distributed compute. Except that Spark allows you to mix/match code with SQL.

Post reply on HN