Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead (2023)

motherduck.com

Post reply on HN
Print view View original

21–30 of 475 posts

Re: Big data is dead (2023)

#21

Post by davedx »

brtkdotse wrote: post #11
blagie wrote: post #4

Overall, I agree with much of this post, but there are several caveats: 1) Mongo is a bad point of reference. The one lesson I've learned is that there is nothing Mongo does which postgresql doesn't do better. Big data solutions aren't nosql / mongo, but usually things like columnar databases, map/reduce, Cassandra, etc. 2) Plan for success 95% of businesses never become unicorns, but that's the goal for most (for th…

> 95% of businesses never become unicorns, but that's the goal for most Is it really the general case or is it just a HN echo chamber meme? My pet peeve is that patterns used by companies that in theory could become global unicorns are mimicked by companies where 5000 paying customers would mean an immense success

It's definitely an echo chamber. Most companies definitely do not want to become "unicorns" - most SME's around the world don't even know what a "unicorn" is, let alone be in an industry/sector where it's possible.

Does a mining company want to become a "unicorn"?

A fish and chip shop?

Even within tech there is an extremely large number of companies whose goals are to steadily increase profits and return them to shareholders. 37 Signals is the posterchild there.

Maybe if you're a VC funded startup then yeah.

Re: Big data is dead (2023)

#22

Post by [deleted] »

notachatbot1234 wrote: post #15
blagie wrote: post #4

Overall, I agree with much of this post, but there are several caveats: 1) Mongo is a bad point of reference. The one lesson I've learned is that there is nothing Mongo does which postgresql doesn't do better. Big data solutions aren't nosql / mongo, but usually things like columnar databases, map/reduce, Cassandra, etc. 2) Plan for success 95% of businesses never become unicorns, but that's the goal for most (for th…

> - Most data isn't big. I can fit data about every person in the world on a $100 Chromebook. (8 billion people * 8 bits of data = 8GB) Nitpick but I cannot help myself: 8 bits are not even enough for a unique integer ID per person, that would require 8 bytes per person and then we are at 60GB already. I agree with pretty much anything else you said, just this stood out as wrong and Duty Calls.

[deleted]

Re: Big data is dead (2023)

#23

Post by threeseed »

blagie wrote: post #4

Overall, I agree with much of this post, but there are several caveats: 1) Mongo is a bad point of reference. The one lesson I've learned is that there is nothing Mongo does which postgresql doesn't do better. Big data solutions aren't nosql / mongo, but usually things like columnar databases, map/reduce, Cassandra, etc. 2) Plan for success 95% of businesses never become unicorns, but that's the goal for most (for th…

> nothing Mongo does which postgresql doesn't do better

a) It has a built-in and supported horizontal scalability / HA solution.

b) For some use cases e.g. star schemas it has significantly better performance.

> Big data solutions aren't nosql

Almost all big data storage solutions are NoSQL.

Re: Big data is dead (2023)

#24

Post by geertj »

I agree with the article that most data sets comfortably fit into a single traditional DB system. But I don’t think that implies that big data is dead. To me big data is about storing data in a columnar storage format with a weak schema, and using a query system based on partitioning and predicate push down instead of indexes. This allows the data to be used in an ad-hoc way by data science or other engineers to answer questions you did not have when you designed the system. Most setups would be relatively small, but could be made to scale relatively well using this architecture.

Re: Big data is dead (2023)

#25

Post by threeseed »

gbin wrote: post #10

IMHO the main driver for big data was company founders egos. Of course your company will explode and will be a planet scale success!! We need to design for scale! This is really a tragic mistake while your product only needs one SQLite DB until you reach series C.... All the energy should be focused on the product, not its scale yet.

No. Big data was driven by people who had big data problems.

It started with Hadoop which was inspired by what existed at Google and became popular in enterprises all around the world who wanted a cheaper/better way to deal with their data than Oracle.

Spark came about as a solution to the complexity of Hive/Pig etc. And then once companies were able to build reliable data pipelines we started to see AI being able to be layered on top.

Re: Big data is dead (2023)

#26

Post by Shrezzing »

This is a quite good allegory for the way AI is currently discussed (perhaps the outcome will be different this time round). Particularly the scary slide[1] with the up-and-to-the-right graph, which is used in a near identical fashion today to show an apparently inevitable march of progress in the AI space due to scaling laws.

[2]https://motherduck.com/_next/image/?url=https%3A%2F%2Fweb-as...

Re: Big data is dead (2023)

#27

Post by WesolyKubeczek »

silvestrov wrote: post #7
ahartmetz wrote: post #3

I guess that hype cycle ended at the plateau of being dead. A not uncommon outcome in this incredibly fashion-driven industry.

It has just been rebranded as AI. AI also use all the data, just with a magick neural network to figure out what it all means.

Given how often it hallucinates, it should be rebranded to "high data".

Re: Big data is dead (2023)

#28

Post by teleforce »

Not dead, it's just having it's winter time not unlike AI winter and once it has its similar "chatbot" moment, all will be well.

My take on the killer application is the climate change for example earthquakes monitoring. For a case study China has just finished building world's largest earthquake monitoring system with the cost of around USD1 Billion across the country with 15K stations [1]. Somehow at the moment is just monitoring existing earthquakes. But let's say there is a big data analytics technique can reliably predicts impending earthquake within a few days, that can probably safe many people and China now still hold the records of the largest mortality and casualty numbers due to earthquakes. Is it probable, the answer is a positive yes based on our work and initial results it's already practical but in order to do that we need integration with comprehensive in-situ IoT networks with regular and frequent data sampling similar to that of China.

Secondly, China also has the largest radio astronomy telescopes and these telescopes together with other radio telescopes collaborate in real-time through e-VLBI to form a virtual giant radio telescopes as big as the earth to monitor distance stars and galaxy. This is how the black hole got its first image but at the time due to logistics one of the telescope remote disks cannot be shipped to the main processing centers in US [2]. At that moment they are not using real-time e-VLBI onky VLBI, and it tooks them several months just to get the complete sets of the black holes observation data. With e-VLBI everything is real-time and with automatic processing it will be hours instead of month. These radio telescopes can also be used for other purposes like monitoring climate change in addition to imaging black holes, their data is astronomical pardon the pun [3].

[1] Chinese Nationwide Earthquake Early Warning System and Its Performance in the 2022 Lushan M6.1 Earthquake:

https://www.mdpi.com/2072-4292/14/17/4269

[2] How Scientists Captured the First Image of a Black Hole:

https://www.jpl.nasa.gov/edu/news/2019/4/19/how-scientists-c...

[3] Alarmed by Climate Change, Astronomers Train Their Sights on Earth:

https://www.nytimes.com/2024/05/14/science/astronomy-climate...

Re: Big data is dead (2023)

#29

Post by threeseed »

brtkdotse wrote: post #11
blagie wrote: post #4

Overall, I agree with much of this post, but there are several caveats: 1) Mongo is a bad point of reference. The one lesson I've learned is that there is nothing Mongo does which postgresql doesn't do better. Big data solutions aren't nosql / mongo, but usually things like columnar databases, map/reduce, Cassandra, etc. 2) Plan for success 95% of businesses never become unicorns, but that's the goal for most (for th…

> 95% of businesses never become unicorns, but that's the goal for most Is it really the general case or is it just a HN echo chamber meme? My pet peeve is that patterns used by companies that in theory could become global unicorns are mimicked by companies where 5000 paying customers would mean an immense success

HN is the worst echo chamber around.

Obsessed with this "you must use PostgreSQL for every use case" nonsense.

And that anyone who actually has unique data needs is simply doing it for their resume or are over-engineering.

Re: Big data is dead (2023)

#30

Post by MOARDONGZPLZ »

davedx wrote: post #19
blagie wrote: post #4

Overall, I agree with much of this post, but there are several caveats: 1) Mongo is a bad point of reference. The one lesson I've learned is that there is nothing Mongo does which postgresql doesn't do better. Big data solutions aren't nosql / mongo, but usually things like columnar databases, map/reduce, Cassandra, etc. 2) Plan for success 95% of businesses never become unicorns, but that's the goal for most (for th…

> 2) Plan for success 95% of businesses never become unicorns, but that's the goal for most (for the 5% which do). If you don't plan for it, you won't make it. That's exactly what every architecture astronaut everywhere says. In my experience it's completely untrue, and actually "planning for success" more often than not causes huge drags on productivity, and even more important for startups, on agility. Because peop…

In my experience the drag caused from the thinking to plan for scalability early has been so much greater than the effort to rearchitect things when and if the company becomes a unicorn that one is significantly more likely to become a unicorn if they simply focus on execution and very fast iteration and save the scalability until it’s actually needed (and they can hire a team of whomever to effect this change with their newly minted unicorn cachet).
Post reply on HN