Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead (2023)

motherduck.com

Post reply on HN
Print view View original

31–40 of 475 posts

Re: Big data is dead (2023)

#31

Post by newaccount74 »

davedx wrote: post #19
blagie wrote: post #4

Overall, I agree with much of this post, but there are several caveats: 1) Mongo is a bad point of reference. The one lesson I've learned is that there is nothing Mongo does which postgresql doesn't do better. Big data solutions aren't nosql / mongo, but usually things like columnar databases, map/reduce, Cassandra, etc. 2) Plan for success 95% of businesses never become unicorns, but that's the goal for most (for th…

> 2) Plan for success 95% of businesses never become unicorns, but that's the goal for most (for the 5% which do). If you don't plan for it, you won't make it. That's exactly what every architecture astronaut everywhere says. In my experience it's completely untrue, and actually "planning for success" more often than not causes huge drags on productivity, and even more important for startups, on agility. Because peop…

The biggest problem with planning for scale is that engineers often have no idea what problems they will actually run into when they scale and they build useless shit that slows them down and doesn't help later at all.

I've come to the conclusion that the only strategy that works reliably is to build something that solves problems you have NOW rather than trying to predict the future.

Re: Big data is dead (2023)

#32

Post by kmarc »

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB

I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and many of them came up with astonishing tens of thousands of dollars yearly.

The winner of course was the guy who understood that 6TiB is what 6 of us in the room could store on our smart phones, or a $199 enterprise HDD (or three of them for redundancy), and it could be loaded (multiple times) to memory as CSV and simply run awk scripts on it.

I am prone to the same fallacy: when I learn how to use a hammer, everything looks like a nail. Yet, not understanding the scale of "real" big data was a no-go in my eyes when hiring.

Re: Big data is dead (2023)

#33

Post by maayank »

zurfer wrote: post #8

This is not fully correct. Originally big data was defined by 3 dimensions: - Volume (mostly what the author talks about) [solved] - Velocity, how fast data is processed etc [solved, but expensive] - Variety [not solved] Big Data today is not: I don't have enough storage or compute. It is: I don't have enough cognitive capacity to integrate and make sense of it.

What do you mean by 'variety'?

Re: Big data is dead (2023)

#34

Post by amenhotep »

notachatbot1234 wrote: post #15
blagie wrote: post #4

Overall, I agree with much of this post, but there are several caveats: 1) Mongo is a bad point of reference. The one lesson I've learned is that there is nothing Mongo does which postgresql doesn't do better. Big data solutions aren't nosql / mongo, but usually things like columnar databases, map/reduce, Cassandra, etc. 2) Plan for success 95% of businesses never become unicorns, but that's the goal for most (for th…

> - Most data isn't big. I can fit data about every person in the world on a $100 Chromebook. (8 billion people * 8 bits of data = 8GB) Nitpick but I cannot help myself: 8 bits are not even enough for a unique integer ID per person, that would require 8 bytes per person and then we are at 60GB already. I agree with pretty much anything else you said, just this stood out as wrong and Duty Calls.

Sure it is. You just need a one to one function from person to [0, eight billion]. Use that as your array index and you're golden. 8 GB is overkill, really, you could pack some boolean datum like "is over 18" into bits within the bytes and store your database in a single gigabyte.

Writing your mapping function would be tricky! But definitely theoretically possible.

Re: Big data is dead (2023)

#35

Post by L-four »

iraqmtpizza wrote: post #17
notachatbot1234 wrote: post #15

Earlier quoted context omitted.

> - Most data isn't big. I can fit data about every person in the world on a $100 Chromebook. (8 billion people * 8 bits of data = 8GB) Nitpick but I cannot help myself: 8 bits are not even enough for a unique integer ID per person, that would require 8 bytes per person and then we are at 60GB already. I agree with pretty much anything else you said, just this stood out as wrong and Duty Calls.

meh. memory address is the ID

Airline booking numbers used to just be the sector number of your booking record on the mainframes HDD.

Re: Big data is dead (2023)

#36

Post by Cyberdog »

silvestrov wrote: post #7
ahartmetz wrote: post #3

I guess that hype cycle ended at the plateau of being dead. A not uncommon outcome in this incredibly fashion-driven industry.

It has just been rebranded as AI. AI also use all the data, just with a magick neural network to figure out what it all means.

Assuming you're serious for a moment, I don't think AI is really a practical tool for working with big data.

- The "hallucination" factor means every result an AI tells you about big data is suspect. I'm sure some of you who really understand AI more than the average person can "um akshually" me on this and tell me how it's possible to configure ChatGPT to absolutely be honest 100% of the time but given the current state of what I've seen from general-purpose AI tools, I just can't trust it. In many ways this is worse than MongoDB just dropping data since at least Mongo won't make up conclusions about data that's not there.

- At the end of the day - and I think we're going to be seeing this happen a lot in the future with other workflows as well - you're using this heavy, low-performance general-purpose tool to solve a problem which can be solved much more performatively by using tools which have been designed from the beginning to handle data management and analysis. The reason traditional SQL RDBMSes have endured and aren't going anywhere soon is partially because they've proven to be a very good compromise between general functionality and performance for the task of managing various types of data. AI is nowhere near as good of a balance for this task in almost all cases.

All that being said, the same way Electron has proven to be a popular tool for writing widely-used desktop and mobile applications, performance and UI concerns be damned all the way to hell, I'm sure we'll be seeing AI-powered "big data" analysis tools very soon if they're not out there already, and they will suck but people will use them anyway to everyone's detriment.

Re: Big data is dead (2023)

#37

Post by dventimi »

Question for the Big Data folks: where do sampling and statistics fit into this, if at all? Unless you're summing to the penny, why would you ever need to aggregate a large volume of data (the population) rather than a small volume of data (a sample)? I'm not saying there isn't a reason. I just don't know what it is. Any thoughts from people who have genuine experience in this realm?

Re: Big data is dead (2023)

#38

Post by Shrezzing »

teleforce wrote: post #28

Not dead, it's just having it's winter time not unlike AI winter and once it has its similar "chatbot" moment, all will be well. My take on the killer application is the climate change for example earthquakes monitoring. For a case study China has just finished building world's largest earthquake monitoring system with the cost of around USD1 Billion across the country with 15K stations [1]. Somehow at the moment is…

I think these examples still loosely fits the author's argument:

> There are some cases where big data is very useful. The number of situations where it is useful is limited

Even though there are some great use-cases, the overwhelming majority organisations, institutions, and projects will never have a "let's query ten petabytes" scenario that forces them away from platforms like Postgres.

Most datasets, even at very large companies, fit comfortably into RAM on a server - which is now cost-effective, even in the dozens of terabytes.

Re: Big data is dead (2023)

#39

Post by fuzzy2 »

newaccount74 wrote: post #31
davedx wrote: post #19

Earlier quoted context omitted.

> 2) Plan for success 95% of businesses never become unicorns, but that's the goal for most (for the 5% which do). If you don't plan for it, you won't make it. That's exactly what every architecture astronaut everywhere says. In my experience it's completely untrue, and actually "planning for success" more often than not causes huge drags on productivity, and even more important for startups, on agility. Because peop…

The biggest problem with planning for scale is that engineers often have no idea what problems they will actually run into when they scale and they build useless shit that slows them down and doesn't help later at all. I've come to the conclusion that the only strategy that works reliably is to build something that solves problems you have NOW rather than trying to predict the future.

Exactly this. Not only would they not know the tech challenges, they also wouldn’t know the business/domain challenges.

Re: Big data is dead (2023)

#40

Post by oli5679 »

BigQuery has a generous 1TB/month free tier, and $6/tb afterwards. If you have small data, it's a pragmatic option, just make sure to use partitioning and sensible query patterns to limit the number of full-data scans, as you approach 'medium data' region.

There are some larger data-sizes, and query patterns, where either BigQuery Capacity compute pricing, or another vendor like Snowflake, becomes more economical.

https://cloud.google.com/bigquery/pricing

    BigQuery offers a choice of two compute pricing models for running queries:

    On-demand pricing (per TiB). With this pricing model, you are charged for the number of bytes processed by each query. The first 1 TiB of query data processed per month is free.

    Queries (on-demand) - $6.25 per TiB - The first 1 TiB per month is free.
Post reply on HN