I'm pleased to see SurrealDB making progress. I look forward to the distributed comparisons. Can you also report TPC benchmarks? Also, do you have any write ups on your correctness- and regression testing approach?
Benchmarking SurrealDB 3.x vs. Postgres, Mongo, Neo4j and Redis (With Fsync)
21–30 of 35 posts
Re: Benchmarking SurrealDB 3.x vs. Postgres, Mongo, Neo4j and Redis (With Fsync)
#22> MongoDB still leads on single-record writes, while SurrealDB is ~1.3× faster on reads
I greatly appreciate when a vendor is willing to run the test and publish unfavorable information, even if it's only in one benchmark category.
Re: Benchmarking SurrealDB 3.x vs. Postgres, Mongo, Neo4j and Redis (With Fsync)
#23I really wish they clearly documented the parameters used by each of the databases (or are we expected to dig those out of the .rs sources somehow?), and actual versions used (saying "Postgres" is ambiguous, it could be 14 or 18 - presumably it's 18, but who knows). For example, I see they do this for Postgres: `let max_wal_gb = (shared_buffers_gb).clamp(2, 16);` 2-16GB of WAL is not a lot, but I have no idea how lar…
We definitely should make that clearer in the docs (thanks for highlighting this). The Postgres image used is Postgres 17. On the parameters, the relational tests use 5 million records per test. The exceptions are the key-value category, which uses 15 million records, and the embedded category, which uses 1 million records. The same dataset shape, workload, harness, and hardware are used across the engines being comp…
Can you push the results in a benchmark repo under the GitHub account linked from the parent Readme of the project?
The full transparency would be very helpful to know where these strengths are coming from which at a glance look to be multi-threaded in-memory processing.
Re: Benchmarking SurrealDB 3.x vs. Postgres, Mongo, Neo4j and Redis (With Fsync)
#24I wish they would have compared with Google’s spanner db. I’ve found this database to be the fastest, multi region, globally distributed database by far.
Re: Benchmarking SurrealDB 3.x vs. Postgres, Mongo, Neo4j and Redis (With Fsync)
#25Earlier quoted context omitted.
I disagree. Current posture makes it nearly impossible for European companies to build software centric open source businesses.
Why do you think the OSI should be concerned with European companies and their business models? Why would European companies have a more difficult time building software centric open source businesses than companies in other countries?
Why should European companies (or anyone else) be concerned with definitions made up by OSI?
Re: Benchmarking SurrealDB 3.x vs. Postgres, Mongo, Neo4j and Redis (With Fsync)
#26So, just use PostgreSQL? 50% faster write at cost of 25% slower reads (which usually are prevailing workload) doesn't warrant moving into far smaller ecosystem
Not in anyway related with the surreal folks. I think it is not so clear cut. I mean, the multi-model nature it is pretty neat. Yes, you can use pgvector on PostgreSQL, but here you also have native graph support. If you want to have both you need to also add something like apache AGE, but arguably that is also a small ecosystem (at least IMHO as I never heard it until I actually started looking for Neo4J alternative…
> ...add something like apache AGE, but arguably that is also a small ecosystem (at least IMHO as I never heard it until I actually started looking for Neo4J alternatives)
I appreciate having the option, but for most use cases on Pg, you are better off just using JOINs or switch to Neo4j for your graph workloads. I switched some workloads back to using different approaches of approximating "connectedness" in Pg (e.g. using Jaccard similarity)
If you do go down this route, the easiest way to get coding agents to figure out AGE is actually their regressions SQL tests: https://github.com/apache/age/tree/master/regress/sql
This has a lot of examples for the agent to know what will/won't work with AGE versus Neo4j Cypher.
Re: Benchmarking SurrealDB 3.x vs. Postgres, Mongo, Neo4j and Redis (With Fsync)
#27Note to others, since it wasn't obvious to me at first: their database is proprietary.
Regarding the benchmarks themselves, they would've been more interesting with reproducible steps and more hardware variety.
Re: Benchmarking SurrealDB 3.x vs. Postgres, Mongo, Neo4j and Redis (With Fsync)
#28Note to others, since it wasn't obvious to me at first: their database is proprietary. Regarding the benchmarks themselves, they would've been more interesting with reproducible steps and more hardware variety.
All of our releases thus far are source available, and will be open source 4 years from the release date.
To your point on the benchmarks, this is something we can improve on documenting and will do so for our next release.
Re: Benchmarking SurrealDB 3.x vs. Postgres, Mongo, Neo4j and Redis (With Fsync)
#29Note to others, since it wasn't obvious to me at first: their database is proprietary. Regarding the benchmarks themselves, they would've been more interesting with reproducible steps and more hardware variety.
All of our releases thus far are source available, and will be open source 4 years from the release date. To your point on the benchmarks, this is something we can improve on documenting and will do so for our next release.
> All of our releases thus far are source available, and will be open source 4 years from the release date.
So, the current up-to date version is proprietary.
Re: Benchmarking SurrealDB 3.x vs. Postgres, Mongo, Neo4j and Redis (With Fsync)
#30SQLite benchmarks are deeply flawed. Somebody has proven that it can sustain 4M queries per second on bare metal, 1.5M qps on a cloud VM, and it happened over 7 years ago. Therefore I do not believe any number from SurrealDB.
https://use.expensify.com/blog/scaling-sqlite-to-4m-qps-on-a...