Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Databases in 2021: A Year in Review

ottertune.com

Post reply on HN
Print view View original

111–120 of 136 posts

Re: Databases in 2021: A Year in Review

#111

Post by PhoenixReborn »

throwDec21 wrote: post #41

I'm just surprised that in 2021 BigQuery isn't more popular. I thought it would be top 10 by now, I moved to GCP because of it but feel like I'm the only one.

Because BQ is great for the ETL/data warehouse/BI use case but is terrible for online applications. I tried using BQ as the backing store for an online analytics application back in late 2019, and it was so much worse than using Clickhouse/Druid/Pinot for the same use case. IDK how much that has changed since, but I'm not too terribly surprised that it isn't higher.

Re: Databases in 2021: A Year in Review

#112

Post by PhoenixReborn »

criticaltinker wrote: post #22

Databases in 2030: SQL DB finally succumbs to Graph DB as #1 Does anyone else feel like a caveman when modeling a many to many relationship in a normalized schema, and then querying via SQL? I’m surprised graph DBs aren’t more popular for this reason alone. Maybe it’s a far fetched dream, but perhaps a graph frontend can be slapped onto the Postgres backend.

Have you looked at Hasura for the second question (graph frontend + relational backend)? That's basically GraphQL on top of Postgres.

As for the first question - I've tried using Neo4j and ArangoDB for relatively large-scale graph querying (1-2TB of data) and both couldn't hold a candle to Postgres or MySQL in terms of query performance for cost. Neo requires you to store most of your data in memory and Arango isn't great for cross-shard querying.

Unless there's some major new graph DB that comes out in the next few years I would still bet on relational being dominant in 2030.

Re: Databases in 2021: A Year in Review

#113

Post by _vvhw »

ryanworl wrote: post #108
_vvhw wrote: post #99

What are the distributed options for Postgres? What mechanisms are available to make it highly available i.e. with a distributed consensus protocol for strict serializability when failing over the primary? How do people typically deploy Postgres as a cluster? 1. Async replication tolerating data loss from slightly stale backup after a failover? 2. Sync replication tolerating downtime during manual failover? 3. Distri…

I think you'd still need to change the core of the database to avoid stale reads when an old primary and client are partitioned away from the new primary, or force all client communication through a proxy smart enough to contact a quorum of replicas to ensure the current primary is still the primary during transaction begin and commit.

Ah yes, good point!

I was assuming in both cases of manual failover that the operator would have to have some way of physically shutting down the old primary, then starting it again only as a backup that doesn't reply to clients. Alternatively, the cluster would need to remain unavailable if any node is partitioned.

But none of this is really very practical when compared to a consensus protocol (or R/W quorums) and distributed database. I'm genuinely curious how people solve this with something like Postgres. Or is it perhaps something that isn't much worried about?

Re: Databases in 2021: A Year in Review

#115

Post by howdydoo »

apavlo wrote: post #102
tayo42 wrote: post #37

Wow I kind of feel like I'm reading about Javascript frameworks. I don't recognize any of the dbs or companies/projects. Didn't realize the db world was so busy

If you want to be even more overwhelmed, see my encyclopedia of database systems: https://dbdb.io/

Which database do you use to store your database of databases?

Re: Databases in 2021: A Year in Review

#116

Post by AtlasBarfed »

_vvhw wrote: post #99

What are the distributed options for Postgres? What mechanisms are available to make it highly available i.e. with a distributed consensus protocol for strict serializability when failing over the primary? How do people typically deploy Postgres as a cluster? 1. Async replication tolerating data loss from slightly stale backup after a failover? 2. Sync replication tolerating downtime during manual failover? 3. Distri…

I can't see how #3 scales under any write load unless you have no joins.|

Well, unless each node has a complete copy of the data?

Re: Databases in 2021: A Year in Review

#117

Post by cam0 »

apavlo wrote: post #103
criticaltinker wrote: post #22

Databases in 2030: SQL DB finally succumbs to Graph DB as #1 Does anyone else feel like a caveman when modeling a many to many relationship in a normalized schema, and then querying via SQL? I’m surprised graph DBs aren’t more popular for this reason alone. Maybe it’s a far fetched dream, but perhaps a graph frontend can be slapped onto the Postgres backend.

> Databases in 2030: SQL DB finally succumbs to Graph DB as #1 Graph databases will not overtake relational databases in 2030 by marketshare. Bookmark this comment. Reach out to me in 2030. If I'm wrong, I will replace my official CMU photo with one of me wearing a shirt that says "Graph Databases Are #1". I will use that photo until I retire, get fired, or a former student stabs me.

Not a fan of graph dbs? Surprised the $325m round for Neo4j didn't make your funding paragraph.

https://techcrunch.com/2021/06/17/neo4j-series-f/

Re: Databases in 2021: A Year in Review

#118

Post by jbergens »

throwDec21 wrote: post #41

I'm just surprised that in 2021 BigQuery isn't more popular. I thought it would be top 10 by now, I moved to GCP because of it but feel like I'm the only one.

I'm not on GCP but the one I would like to try is Spanner or Cloud Spanner.

I think more scalable systems will continue to gain market share. It will be interesting to see if PlanetScale, CockroachDb or some other actually becomes a big player.

Re: Databases in 2021: A Year in Review

#119

Post by jbergens »

PhoenixReborn wrote: post #112
criticaltinker wrote: post #22

Databases in 2030: SQL DB finally succumbs to Graph DB as #1 Does anyone else feel like a caveman when modeling a many to many relationship in a normalized schema, and then querying via SQL? I’m surprised graph DBs aren’t more popular for this reason alone. Maybe it’s a far fetched dream, but perhaps a graph frontend can be slapped onto the Postgres backend.

Have you looked at Hasura for the second question (graph frontend + relational backend)? That's basically GraphQL on top of Postgres. As for the first question - I've tried using Neo4j and ArangoDB for relatively large-scale graph querying (1-2TB of data) and both couldn't hold a candle to Postgres or MySQL in terms of query performance for cost. Neo requires you to store most of your data in memory and Arango isn't…

Have you tried TigerGraph?

They say that they scale well. I have not tried any graphdb for prod work yet.

Re: Databases in 2021: A Year in Review

#120

Post by manigandham »

code_biologist wrote: post #87
manigandham wrote: post #51

Earlier quoted context omitted.

I've never seen a business actually use them, large or small. Any auditing requirements are usually fed from other sources, like Kafka event streams, files on S3, or a OLAP data warehouse.

How do you set up and feed the warehouse? Temporal-ish tables have been an obvious, simple, and mostly foolproof solution for many of our historical analytics and reporting needs. Bitemporal stuff (enabling edited versions of history) is where things get hairy and I definitely question the utility outside of a dedicated use case.

Most databases have built-in CDC (change data capture) that can be exported. Otherwise the WAL logs can be read with other tooling.

Debezium is a great open-source product for streaming changes from many relational databases: https://debezium.io

Post reply on HN