Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead (2023)

motherduck.com

Post reply on HN
Print view View original

91–100 of 475 posts

Re: Big data is dead (2023)

#91

Post by the_real_cher »

kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

How would six terabytes fit into memory?

It seems like it would get a lot of swap thrashing if you had multiple processes operating on disorganized data.

I'm not really a data scientist and I've never worked on data that size so I'm probably wrong.

Re: Big data is dead (2023)

#92

Post by GuB-42 »

AI is the new "big data". In fact, AI as it is done today is nothing without at least terabytes of data.

What the article talks about is more like a particular type of database architecture (often collectively called "NoSQL") that was a fad a few years ago, and as all fads, it went down. It doesn't mean having lots of data is useless, or that NoSQL is useless, just that it is not the solution to every problem. And also that there is a reason why regular SQL databases have been in use since the 70s: except in specific situation most people don't encounter, they just work.

Re: Big data is dead (2023)

#94

Post by deadbabe »

Something similar will happen with generative AI someday.

AI scientists will propose all sorts of elaborate complex solutions to problems using LLMs, and the dismissive responsive will be “Your problem is solvable with a couple if statements.”

Most people just don’t have problems that require AI.

Re: Big data is dead (2023)

#95

Post by msl09 »

lokimedes wrote: post #49

I was a researcher at the Large Hadron Collider around the time “Big Data” became a thing. We had one of the use cases where analyzing all the data made sense, since it boiled down to frequentist statistics, the more data, the better. Yet even with a global network of supercomputers at our disposal, we funnily figured out that fast local storage was better than waiting for huge jobs to finish. So, surprise, surprise,…

I think that your law of convenience is spot on. One thing that got by talking with commercial systems devs is that they are always under pressure by their clients to make their systems as cheap as possible, reducing the database stored and the size of the computations is one great way to minimize the client's monthly bill.

Re: Big data is dead (2023)

#97

Post by ozkatz »

threeseed wrote: post #23
blagie wrote: post #4

Overall, I agree with much of this post, but there are several caveats: 1) Mongo is a bad point of reference. The one lesson I've learned is that there is nothing Mongo does which postgresql doesn't do better. Big data solutions aren't nosql / mongo, but usually things like columnar databases, map/reduce, Cassandra, etc. 2) Plan for success 95% of businesses never become unicorns, but that's the goal for most (for th…

> nothing Mongo does which postgresql doesn't do better a) It has a built-in and supported horizontal scalability / HA solution. b) For some use cases e.g. star schemas it has significantly better performance. > Big data solutions aren't nosql Almost all big data storage solutions are NoSQL.

> Almost all big data storage solutions are NoSQL.

I think it's important to distinguish between OLAP AND OLTP.

For OLAP use cases (which is what this post is mostly about) it's almost 100% SQL. The biggest players being Databricks, Snowflake and BigQuery. Other tools may include AWS's tools (Glue, Athena), Trino, ClickHouse, etc.

I bet there's a For OLTP "big data", I'm assuming we're talking about "scale-out" distributed databases which are either SQL (e.g. cockroachdb, vitess, etc) SQL-like (Casandra's CQL, Elasticsearch's non-ANSI SQL, Influx' InfluxQL) or a purpose-built language/API (Redis, MongoDB).

I wouldn't say OLTP is "almost all" NoSQL, but definitely a larger proportion compared to OLAP.

Re: Big data is dead (2023)

#98

Post by ImPostingOnHN »

pdimitar wrote: post #62
kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

Blows my mind. I am a backend programmer and a semi-decent sysadmin and I would have immediately told you: "make a ZFS or BCacheFS pool with 20-30% redundancy bits and just go wild with CLI programs, I know dozens that work on CSV and XML, what's the problem?". And I am not a specialized data scientist. But with time I am wondering if such a thing even exists... being a good backender / sysadmin and knowing a lot of…

> But with time I am wondering if such a thing even exists

Check out "data science at the command line":

https://jeroenjanssens.com/dsatcl/

Re: Big data is dead (2023)

#99

Post by CoastalCoder »

vegabook wrote: post #6

my experience is that while data keeps growing at an exponential rate, its information content does not. In finance at least, you can easily get 100 million data points per series per day if you want everything, and you might be dealing with thousands of series. That sample rate, and the number of series, is usually 99.99% redundant, because the eigenvalues drop off almost to zero very quickly after about 10 dimensio…

Could you point to something explaining that eigenvalue / dimensions topic?

It sounds interesting, but it's totally new to me.

Re: Big data is dead (2023)

#100

Post by zurfer »

maayank wrote: post #33
zurfer wrote: post #8

This is not fully correct. Originally big data was defined by 3 dimensions: - Volume (mostly what the author talks about) [solved] - Velocity, how fast data is processed etc [solved, but expensive] - Variety [not solved] Big Data today is not: I don't have enough storage or compute. It is: I don't have enough cognitive capacity to integrate and make sense of it.

What do you mean by 'variety'?

the other comments get it.

It means that data comes in a ton of different shapes with poorly described schemas (technically and semantically).

From the typical CSV export out of an ERP system to a proprietary message format from your own custom embedded device software.

Post reply on HN