[deleted]
Do you even need a database?
261–270 of 311 posts
Re: Do you even need a database?
#262I love this article as it shows how fast computers really are. There is one conclusion that I do not agree with. Near the end, the author lists cases where you will outgrow flat files. He then says that "None of these constraints apply to a lot of applications." One of the constraints is "Multiple processes need to write at the same time." It turns out many early stage products need crons and message queues that exec…
Yeah, files as a database is fun, but I find you ultimately reinvent the wheel when sqlite is pretty battle tested, free and easier to get right or scale up.
I can't talk though because I actually find myself doing this a lot
Re: Do you even need a database?
#263so the article benchmarks speed, but I don't really think most people are making the choice based on speed. Robustness is really the driving force (well at least for me). So for CRUD operations I have way more confidence with something like SQLite than using files. Files are workable in many situations, but much easier if you default to things like SQLite and use files only when you think there is a specific advantage.
Re: Do you even need a database?
#264[dead]
Re: Do you even need a database?
#265so the article benchmarks speed, but I don't really think most people are making the choice based on speed. Robustness is really the driving force (well at least for me). So for CRUD operations I have way more confidence with something like SQLite than using files. Files are workable in many situations, but much easier if you default to things like SQLite and use files only when you think there is a specific advantag…
It's like the old words of wisdom say, the fastest way to manage your data is to pipe it to /dev/null
Re: Do you even need a database?
#266[deleted]
Re: Do you even need a database?
#267There's a whole thing this days about building solvers (e.g. SAT or Ising) out of exotic hardware that does compute in memory. A while back I wondered if one could leverage distributed DB logic to build solvers for massive problems, something like compute in DB.
Re: Do you even need a database?
#268For anything that starts as a side project, SQLite covers it. The moment you need to ask this question the answer is usually "not yet." Ship first, migrate when it actually hurts
Re: Do you even need a database?
#269You need databases if you need any kind of atomicity. Doing atomic writes is extremely fragile if you are just on top of the filesystem. This is also why many databases have persistence issues and can easily corrupt on-disk data on crash. Rocksdb on windows is a very simple example a couple years back. It was regularly having corruption issues when doing development with it.
> Doing atomic writes is extremely fragile if you are just on top of the filesystem. This is not true, at least in Linux. pwritev2(fd, iov, iovcnt, offset, RWF_ATOMIC); The requirements being that the write must be block-aligned and no larger than the underlying FS's guaranteed atomic write size
[deleted]
Re: Do you even need a database?
#270My personal toolbox’s section for databases has three items: files (usually JSON), SQLite, Postgres. I haven’t and I doubt I will ever run into a situation where none of those fit. They exist, I’m sure, but I’m too general purpose to ever need anything else.