I’ve often wanted a ‘database construction set’ in that I want a particular set of features and data structures and would be willing to throw out everything I don’t need for performance.
Never write a database, even if you want to, even if you think you should
31–40 of 56 posts
Re: Never write a database, even if you want to, even if you think you should
#32Don’t take Twitter advice… this post does nothing but discourage play and experimentation.
It's just a twitter take, overcompressed and lacking nuance. By all means write a toy database for learning purposes, but in 99+% of cases you should probably then turn around and use the ones with tens of thousands of engineering hours poured into their correctness properties when you want to put something into production.
Re: Never write a database, even if you want to, even if you think you should
#33From an engineering perspective you should not think you can solve
whatever issue you have by writing your own database.
Writing one is one of the harder tasks in computer science.
You should write a database system if you want to, for learning, fun, to try out a lot of different approaches, and see how and why things are solved in other DBMS systems.
You can even try using it on small fun projects.
Dont ever think of putting into production on something important. It will come back and haunt you in the most undesirable ways.
Re: Never write a database, even if you want to, even if you think you should
#34Don’t take Twitter advice… this post does nothing but discourage play and experimentation.
The tweet is part of a conversation about building a database for your business, it has nothing to do with play and experimentation for the sake of learning
Except it's a discussion where the author is in reality bragging about how their product is only viable because they wrote their own database rather than using an off the shelf one, and then keeps repeating this "don't write a database" bit in every other tweet as a lame joke.
Re: Never write a database, even if you want to, even if you think you should
#35Always take commands about what to work on / not work on from random Xitterers.
Re: Never write a database, even if you want to, even if you think you should
#36Earlier quoted context omitted.
I don't think most people would disagree with that. If you want to build a database, for curiosity, or to solve a particular problem, or just because it seems fun, go ahead. But know that making a real production database tends to be harder than it looks, and so tends to be all-consuming, and a team that depends on a custom database tends to become a database team rather than an anything else team. That could be the…
> But know that making a real production database tends to be harder than it looks, and so tends to be all-consuming, and a team that depends on a custom database tends to become a database team rather than an anything else team. The original statement does not include any caveat or constraint, and is an absolute statement that does not depend on specifics. The statement is not "Never write a custom database as a sub…
It is a statement directed at a readership of — and therefore assuming it will only be read by — application developers, rather than systems programmers; and so the context of "Never write a custom database as a subproject of any other project" is implicit.
Also, you're reacting to an incomplete quote. The tweet is:
> "Never write a database. Even if you want to, even if you think you should. Resist. Never write a database. Unless you have to write a database. But you don't."
There are precisely two cases where you "have to write a database":
1. when you want to learn how databases are architected, and educational material on the ground is wholly inadequate (which it is!); and
2. when you discover, under a production workload, that no existing DBMS sits in exactly the right part of DBMS configuration-space to meet your scaling needs — and nor can any existing DBMS even be modified to reach that point in DBMS configuration-space — and nor can you modify the design of your application to scale in a different way; such that the only possible way to address your scaling challenges is to write an entirely novel DBMS with an entirely novel architecture.
I can only think of two times #2 has happened in recent memory: Amazon with Dynamo (the core of S3 and DynamoDB); and Google with Dremel (the core of BigQuery.) Both projects have resulted in many research papers, and later copycat DBMSes (Dynamo → Cassandra, Riak, BigTable, etc; Dremel → Redshift, Snowflake, etc.) If you don't think your DB architecture is novel enough to result in that kind of response, then it's probably not something that requires "writing a database" — it's probably instead something you can do as a hack on top of some existing database. (Think: the way Citus/Timescale/Greenplum build on Postgres.)
Re: Never write a database, even if you want to, even if you think you should
#37Of course, in production, it is really hard to find a use case that isn’t served ‘well enough’ by existing database technology. But: almost ever should you say never. What better way to learn indexing tradeoffs than to write your own database? What existed before Redis? What existed before LevelDB? What existed before Git? Sure, write a database if you want. But be humble about the effort involved and reliability you…
BerkeleyDB existed before Redis, LevelDB, and Git and was widely used as a database. SVN originally used BDB as its storage layer (but switched when people complained it was too hard to build BDB). https://en.wikipedia.org/wiki/Berkeley_DB contains people-decades of hard-won experience on how to build a simple database system. I would consider SQLite, itself 20+ years old now, the intellectual heir of BDB. It was as…
BerkeleyDB also improved on things like dbm¹, ndbm, qdbm and cdb.
> I would consider SQLite, itself 20+ years old now, the intellectual heir of BDB.
Indeed. I've worked on projects that only use the btree part of SQLite. If all you need is a btree it's a good place to start.
Re: Never write a database, even if you want to, even if you think you should
#38Earlier quoted context omitted.
> But know that making a real production database tends to be harder than it looks, and so tends to be all-consuming, and a team that depends on a custom database tends to become a database team rather than an anything else team. The original statement does not include any caveat or constraint, and is an absolute statement that does not depend on specifics. The statement is not "Never write a custom database as a sub…
The tweet doesn’t but the context of the tweet is a conversation about building a custom database to power a business.
Well, the company I worked at did, and it worked out just fine. Not even that many people involved, like 3-4, the main architect and coder had never done a DB before (but is a genius anyway). For the scope and within that domain (not relational, heavy writes), the performance smokes anything out there. In production for over a decade now.
Re: Never write a database, even if you want to, even if you think you should
#39[deleted]
Re: Never write a database, even if you want to, even if you think you should
#40Don’t take Twitter advice… this post does nothing but discourage play and experimentation.
The tweet is part of a conversation about building a database for your business, it has nothing to do with play and experimentation for the sake of learning
Link is not working for me. Title uses word `never`.