61–70 of 172 posts
Earlier quoted context omitted.
We are using text files. How many programs in a typical Unix installation need to communicate to a database? Next to none. How many are communicating via text files? Almost all of them.
The file system is just a shitty db. Why does it matter whether it writes to a file system or to sqlite3?
I'm glad he loves databases, databases have been the bane of my existence. However, the torment they have given me has also lead to a similar fascination - and now I'm writing my own database! So I've become very familiar with the topics he writes on, and they are very good points for anybody interested in the subject. Why would I write my own database? Because databases are hard, and I am determined to make them eas…
Sure you can. In fact, almost every eventually consistent database is built on a collection of strongly consistent single systems. Eventually consistent systems are even frequently built on top of consistent distributed systems, and some of the largest infrastructure on the planet works just like that.
I hate databases. People tend to have way too much faith in them (or their surrounding marketing), and thus make poor database choices that don't actually fit the shape of their data. Persistence is fundamentally the programmer's responsibility; a magic box behind a socket can't design it for you. Most applications I've seen wouldn't even need a database, but apparently a lot of programmers are conditioned into belie…
The problem is this -- if your developers are good enough to understand all the operating system nuances of handling IO -- (how does dirty page writeback works, buffer cache vs page cache, opening file with O_DIRECT, what does append only mode do exactly to the file, how do you use append-only mode effectivelt, what operations are atomic on the file system, how does mmap function work, what about fadvise, etc. etc.), they are probably smart enough to not use the database and write their own storage layer, OR they are also smart enough to read through the fine print and pick a good database.
If they don't know those things, they will want to reach and pick a database off the shelf, so to speak. They might pick a bad database. Because they might not know how to read the fine print, how to benchmark, how to benchmark failure, how to not read into the marketing bullshit "But it said Webscale on the front page, how come it ate my data, I don't understand...".
Unfortunately quite often databases are treated like black boxes and they are chosen not necessarily becuase of technical merits. Sometimes it is stupid marketing sometimes it is golfware ("CEO spent time playing golf with an Oracle saleman. Salesman lost just enough, but in the end everyone is forced to use Oracle from now on).
I hate databases. People tend to have way too much faith in them (or their surrounding marketing), and thus make poor database choices that don't actually fit the shape of their data. Persistence is fundamentally the programmer's responsibility; a magic box behind a socket can't design it for you. Most applications I've seen wouldn't even need a database, but apparently a lot of programmers are conditioned into belie…
* will it be read or write heavy?
* by what data/key will I access the persistent data?
Most commonly for the type of work that I do, it's write once - read N number of times. What I end up doing then is just to keep the data in-mem in a hash table while writing it async to disk. Concurrent reads hit the hash table while the disk write is being performed. The file(s) are indexed by using a hash of the data itself, multiple ways to to lookups can also be done by using softlinks. Hashing lookup-keys are almost always prefered because of possible locale/security reasons (makes it harder for you to implement an accidental LFI if you only deal with base36) If needed and/or depending on access patterns, I also keep the N most accessed items in a hash table in-mem over time. The API for this usually ends up as two functions, Get and Put.
The amount of code for doing this is a couple of hundred lines at most and it's complexity I've found by experience to be preferable to using a db for the type of work that I do. YMMV.
Earlier quoted context omitted.
When you say "NoSQL-Koolaid free", do you mean "focuses on Relational" or "treats the topic objectively/academically"?
Not OP, but the latter. The author talks both about data models (relational, document, graph-based, with some nice historical tidbits about IMS and CODASYL) and storage models (from B-trees to SSTables, passing through bitmap encoding and everything else). It is an amazing book so far, and I'm pretty excited for the rest of it to be written. It's the best text I've read so far on databases that gives all the referenc…
Earlier quoted context omitted.
Postgres, and MSSQL if you can afford it, are the Swiss Army Knives of database systems. And I mean that in a good way.
For a second I thought you said MySQL and was a bit puzzled. I know indirectly that MSSQL is quite good database, but since you are putting Open Source and proprietary database next to each other it makes me wonder why you picked MSSQL and for example skipped its competitor such as Oracle. This is an honest question, I know very little about MSSQL or Oracle so I'm curious why you picked this one. I know that at least…
"The study of databases intersects almost every topic in computer science" - I've heard this before, especially for Compilers. But it has been false for a long time, CS is far more diverse now. For example, how do Databases intersect AI/Machine Learning/Computer Vision, Computer Graphics, Numerics/Simulation, Robotics, Bioinformatics, Computer Architecture or Cryptography?
Actually it has been getting more true for a long time. With computers in nearly every aspect of society we have more data than ever to sift through. AI/ML is a big data problem. Numerics and simulations become more effective with large amounts of data. Instead of simulating a result from 1 data point, try 100s, 1000s or millions or data points.
Earlier quoted context omitted.
Postgres, and MSSQL if you can afford it, are the Swiss Army Knives of database systems. And I mean that in a good way.
For a second I thought you said MySQL and was a bit puzzled. I know indirectly that MSSQL is quite good database, but since you are putting Open Source and proprietary database next to each other it makes me wonder why you picked MSSQL and for example skipped its competitor such as Oracle. This is an honest question, I know very little about MSSQL or Oracle so I'm curious why you picked this one. I know that at least…
Oracle is something you only use when you have reached some problem that only Oracle can solve. Even then, I would still try to work around Oracle. It is expensive and requires a large amount of tuning to make work well.
Oh, and pretend MySQL doesn't exist.
I'm glad he loves databases, databases have been the bane of my existence. However, the torment they have given me has also lead to a similar fascination - and now I'm writing my own database! So I've become very familiar with the topics he writes on, and they are very good points for anybody interested in the subject. Why would I write my own database? Because databases are hard, and I am determined to make them eas…
Earlier quoted context omitted.
Its pretty funny how right above this comment in the thread suggests the exact opposite. I think "use the right tool for the job" is a notoriously misleading phrase. What does "right" mean, and what is "the job?" If you are at the point where certain databases simply won't work , then of course there's a way to measure this, but most of us are not in this situation. There is operational and mental overhead for each d…
I just don't think you understand the expression. "Right tool for the right job" means from the perspective of the person choosing the tool. There is no "right" answer. And saying "use PostgreSQL for everything" is no different to saying "use C++ for everything". It's dumb and sloppy. Choosing a database is an important decision. Just like choosing an application stack. People should be encouraged to spike different…