Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Databases in 2024: A Year in Review

cs.cmu.edu

Post reply on HN
Print view View original

41–50 of 220 posts

Re: Databases in 2024: A Year in Review

#41

Post by atombender »

The article mentions Greenplum, but it's worth noting that when the code was closed, several of the original developers created an open-source fork, Cloudberry, which seems to be thriving. Cloudberry was accepted into the Apache project this year, and has synced with Postgres 14, whereas the closed-source Greenplum is still stuck on Postgres 12.

The architecture is quite ancient at this point, but I'm not sure it's completely outdated. It's single-master shared-nothing, with shards distributed among replicas, similar to Citus. But the GPORCA query planner is probably the most advanced distributed query planner in the open source world at this point. From what I know, Greenplum/Cloudberry can be significantly faster than Citus thanks to the planner being smarter about splitting the work across shards.

Re: Databases in 2024: A Year in Review

#42

Post by PeterZaitsev »

I think one thing Andy misses about why people were pissed about Elastic and Redis but not as many for MongoDB and some other is their license and size of Contributors Community.

When original license is as restricted as AGPL it is unlikely there is much of embedded use... so less people are impacted in truly catastrophic way

Also if there is no contributor community to speak of... who is going to do the fork ?

I put some thoughts about it in my post about ScyllaDB https://peterzaitsev.com/thoughts-on-scylladb-license-change...

Re: Databases in 2024: A Year in Review

#43

Post by antirez »

nicoritschel wrote: post #40
antirez wrote: post #14

Wow, the reasons why Redis commands API suck in Andy's video (linked in the post) are the weakest ever. It is possible to make a case against the Redis API (I would not agree of course but... it's totally legitimate), but you gotta have stronger arguments than those, particularly if you are a teacher of some kind. Especially: you need to be somewhat fluent in Redis and how developers use Redis in order to understand…

With all due respect, the linked video was pretty fair. It didn't imply not to use Redis, just not as a primary datastore. I don't think folks work with Redis out of fondness for the model, but because it's the least worst datastore for caching, lightweight message broker, and simple realtime things like counters.

Talking about the broken API argument here. Also Redis is particularly useful exactly in other situations compared to what OP says. Leaderboards style use cases with sorted sets are killer applications (super hard to model with SQL) of the data structure server thing. Apparently OP does not understand this and says "simple GET/SET" is what you should use Redis for.

Redis has probabilistic data structures, the ability to implement complex queueing patterns, and so forth. That's where the value is. Otherwise we would still be just with Memcached without caring about Redis. Another killer app was Twitter initial use case (then they used it for pretty much everything): to cache latest N Tweets, using capped lists. I could continue forever.

So OP argoment is flawed IMHO, for the above arguments, not fair. When you talk to students you need to make your homeworks. Really understand the system you are talking and provide a realistic image of it. Then, yes, if you want, criticize it as much as you want, with grounded arguments.

You know what? I re-read this comment and it's embarassing I ever have to write this, because after 15 years of Redis history at such scale and popularity, pretty much everybody that was seriously exposed to Redis knows those stuff. Is tech culture really degraded so much that we have to restate the obvious? Do I really need to explain GET/SET is not exactly where Redis shines after 15 years of half the Internet used all the kind of Redis patterns?

Re: Databases in 2024: A Year in Review

#44

Post by kwillets »

I spent the past year puzzling over the DB market as well, but I don't feel like I'm much closer to understanding it.

It appears that a lot of attention is now directed at the folks doing 100 MB queries, and the high end has moved past everybody's radar. My idea of an exciting product is Ocient, who have skipped over Cloud and gone for hyperscale on-prem hardware. Yellowbrick is also a contender here.

I have a lot of experience with Vertica, and they seem to have gotten stuck in this niche as well, with sales tilted towards big accounts, but less traction in smaller shops, and a difficult road to get a SaaS or similar easy-start offering.

There's a crossover point where self-managed is cheaper than cloud, but nobody seems to have any idea where it is. Snowflake will gladly tell you that your sub-$1M Vertica cluster should be replaced by $10M of sluggish SaaS, and that you are saving money by doing so. These decisions seem more in the realm of psychology or political science.

DHH's cloud exit was a refreshing take on the expense issue, even if it wasn't strictly in the database space -- the cost per VCPU and so forth that he documented is a good start for estimating savings, and he debunked a lot of the "hidden costs" that cloud maximalists claim.

In the business/financial space the biggest news to me was the correction in Snowflake's stock price, which seemed to indicate that investors were finally noticing metrics like price-performance, but they added a little more AI and went back into irrationality.

I'm heavily in favor of DuckDB, Hudi, Iceberg, S3 tables, and the like. Mixing high-end and low-end tools seems like the best strategy (although settling on one high-end DWH has also worked IME), and the low end is getting better and cheaper, squeezing out the mid-range SaaS vendors.

In research I found Goetz Graefe's work in offset-value coding exciting -- he's wired it into query operators in a way that saves a lot of CPU on sorting and joins/aggregation. This is a technique that I've applied favorably in string sorting, and it was discovered in the DB community decades ago but largely forgotten. (This work precedes 2024, but I'm a slow study.)

Re: Databases in 2024: A Year in Review

#45

Post by teej »

codeulike wrote: post #17

Weird how SQL Server and its Azure variants gets no mention. It dominates in certain sectors. DBEngines ranks it third most popular overall https://db-engines.com/en/ranking

Are people choosing SQL Server independently of the Microsoft ecosystem? My understating is that you typically use it because you’re forced to choose a MS product.

Re: Databases in 2024: A Year in Review

#47

Post by tayo42 »

PeterZaitsev wrote: post #42

I think one thing Andy misses about why people were pissed about Elastic and Redis but not as many for MongoDB and some other is their license and size of Contributors Community. When original license is as restricted as AGPL it is unlikely there is much of embedded use... so less people are impacted in truly catastrophic way Also if there is no contributor community to speak of... who is going to do the fork ? I put…

> In the case of Redis, I can only think that people perceive Redis Ltd. as unfairly profiting off others' work since the company's founders were not the system's original creators. An analysis of Redis' source code repository also shows that a sizable percentage of contributions to the DBMS comes from outside the company

He mentions this in "Andy’s Take" section btw

Re: Databases in 2024: A Year in Review

#48

Post by tayo42 »

antirez wrote: post #14

Wow, the reasons why Redis commands API suck in Andy's video (linked in the post) are the weakest ever. It is possible to make a case against the Redis API (I would not agree of course but... it's totally legitimate), but you gotta have stronger arguments than those, particularly if you are a teacher of some kind. Especially: you need to be somewhat fluent in Redis and how developers use Redis in order to understand…

> Andy's video (linked in the post)

Is there a "to long didnt watch" summary any one knows of? I hate videos, but am curious lol

Re: Databases in 2024: A Year in Review

#49

Post by nojito »

antirez wrote: post #16
nojito wrote: post #15

Earlier quoted context omitted.

SQL is king and history has shown non-sql languages are not good which causes many non-sql DBMS's to adopt sql eventually.

Many non-SQL DBs had query languages that were broken Javascript-ish versions of SQL. Of course, this is wrong, and people will eventually adopt SQL instead. But if your data model isn't anything like relational DBs, non-SQL makes a ton of sense. OP seems to miss exactly this, that the Redis query language is shaped on the Redis data model, that is basically alien to the relational model. The idea behind Redis data m…

>But if your data model isn't anything like relational DBs, non-SQL makes a ton of sense. OP seems to miss exactly this, that the Redis query language is shaped on the Redis data model, that is basically alien to the relational model.

Sure...but all roads lead back to SQL eventually. Another recent example also mentioned in the OP is BigTable adopting SQL.

Re: Databases in 2024: A Year in Review

#50

Post by ak_111 »

Wow his database startup that raised 12M died this year after only three years.

If anything this shows how insanely difficult it must be to succeed as a database startup (when was the most recent startup success in this space?), as the founding team is stellar.

On the other hand I am surprised it died this quick and interested to know if they did a proper postmortem. Not only did they raise way more than is needed to survive for three years but the idea is about utilising AI to improve DB performance and I find it hard to imagine they couldn't find more investors to lend them a lifeline with all the AI hype.

Post reply on HN