Big data is dead (2023)

previous discussion: https://news.ycombinator.com/item?id=34694926

Overall, I agree with much of this post, but there are several caveats:

1) Mongo is a bad point of reference.

The one lesson I've learned is that there is nothing Mongo does which postgresql doesn't do better. Big data solutions aren't nosql / mongo, but usually things like columnar databases, map/reduce, Cassandra, etc.

2) Plan for success

95% of businesses never become unicorns, but that's the goal for most (for the 5% which do). If you don't plan for it, you won't make it. The reason to architect for scalability when you have 5 customers is so if that exponential growth cycle hits, you can capitalize on it.

That's not just architecture. To have any chance of becoming a unicorn, every part of the business needs to be planned for now and for later: How do we make this practical / sustainable today? How do we make sure it can grow later when we have millions of customers? A lot of this can be left as scaffolding (we'll swap in [X], but for now, we'll do [Y]).

But the key lessons are correct:

- Most data isn't big. I can fit data about every person in the world on a $100 Chromebook. (8 billion people * 8 bits of data = 8GB)

- Most data is rarely queried, and most queries are tiny. The first step in most big data jobs I've done is taking terabytes of data and shrinking it down to the GB, MB, or oven KB-scale data I need. Caveat: I have no algorithm for predicting what I'll need in the future.

- Cost of data is increasing with regulatory.

my experience is that while data keeps growing at an exponential rate, its information content does not. In finance at least, you can easily get 100 million data points per series per day if you want everything, and you might be dealing with thousands of series. That sample rate, and the number of series, is usually 99.99% redundant, because the eigenvalues drop off almost to zero very quickly after about 10 dimensions, and often far fewer. There's very little reason to store petabytes of ticks that you will never query. It's much more reasonable in many cases to do brutal (and yes, lossy) dimensionality reduction _at ingests time_, store the first few principal components + outliers, and monitor eigenvalue stability (in case some new, previously negligable, factor, starts increasing in importance). It results in a much smaller dataset that is tractable and in many cases revelatory, because it's actually usable.

This is not fully correct.

Originally big data was defined by 3 dimensions:

- Volume (mostly what the author talks about) [solved]

- Velocity, how fast data is processed etc [solved, but expensive]

- Variety [not solved]

Big Data today is not: I don't have enough storage or compute.

It is: I don't have enough cognitive capacity to integrate and make sense of it.

IMHO the main driver for big data was company founders egos. Of course your company will explode and will be a planet scale success!! We need to design for scale! This is really a tragic mistake while your product only needs one SQLite DB until you reach series C.... All the energy should be focused on the product, not its scale yet.