Are people aware of Python object databases like ZODB and Durus? I'm not very familiar with Redis. However, the model used by ZODB and Durus (on disk durable storage, in memory client caches) can be extremely efficient depending on workloads.
Redis: new disk storage to replace VM
11–20 of 24 posts
Re: Redis: new disk storage to replace VM
#12I am curious as to what people who use Redis in production think of these types of changes. Is this alarming or hopeful? Seems like a rather large shift in trade-offs and a whole new set of tuning parameters to play with.
Re: Redis: new disk storage to replace VM
#13I am curious as to what people who use Redis in production think of these types of changes. Is this alarming or hopeful? Seems like a rather large shift in trade-offs and a whole new set of tuning parameters to play with.
Alarming, I'd imagine:
https://groups.google.com/forum/#!topic/redis-db/ZTSm-1w-6AQ
To quote from the end of the post:
so, to sum this up -- after a while, you are stuck with an
in-memory database that you cannot backup, cannot replicate to
a standby machine, and that will eventually consume all memory
and crash (if it does not crash earlier).
conclusion: redis with vm enabled is pretty much unusable, and we
would really not recommend it to anybody else for production use
at the moment. (at least not as a database, it might work better
as a cache.)Re: Redis: new disk storage to replace VM
#14I am curious as to what people who use Redis in production think of these types of changes. Is this alarming or hopeful? Seems like a rather large shift in trade-offs and a whole new set of tuning parameters to play with.
Alarming, I'd imagine: https://groups.google.com/forum/#!topic/redis-db/ZTSm-1w-6AQ To quote from the end of the post: so, to sum this up -- after a while, you are stuck with an in-memory database that you cannot backup, cannot replicate to a standby machine, and that will eventually consume all memory and crash (if it does not crash earlier). conclusion: redis with vm enabled is pretty much unusable, and we would re…
Actually, that is talking about the existing virtual-memory (VM) implementation, which swaps data in and out to disk, and doesn't work so great.
The change being talked about here is all about replacing that exact flakey VM with a more solid disk backed approach
Re: Redis: new disk storage to replace VM
#15Earlier quoted context omitted.
Alarming, I'd imagine: https://groups.google.com/forum/#!topic/redis-db/ZTSm-1w-6AQ To quote from the end of the post: so, to sum this up -- after a while, you are stuck with an in-memory database that you cannot backup, cannot replicate to a standby machine, and that will eventually consume all memory and crash (if it does not crash earlier). conclusion: redis with vm enabled is pretty much unusable, and we would re…
Actually, that is talking about the existing virtual-memory (VM) implementation, which swaps data in and out to disk, and doesn't work so great. The change being talked about here is all about replacing that exact flakey VM with a more solid disk backed approach
I'm sorry, but that's what I meant. I'd be more alarmed than enticed to discover that the current implementation of datasets-larger-than-RAM for my chosen database was considered "flakey", and was going to be swapped out for a green-field approach in the next release.
For reference, this is the blog post that introduced the VM idea: http://antirez.com/post/redis-virtual-memory-story.html
Re: Redis: new disk storage to replace VM
#16So, as a non-Redis user, am I correct in my understanding that an admin may define a maximum delay before write-behind persistence must be attempted? For applications which can afford to lose a few seconds of data in the case of a failure, this seems like a great way to improve latency. More generally, why are there no MySQL (or whatever) engines which offer similar capabilities? Wouldn't it be possible for DB client…
All modern relational databases implement exactly what you've described... transactions are written to a transaction log, which is flushed to disk every few seconds (or whenever you want to guarantee that a txn is durable). Changes to the actual data need not be persisted in a timely manner, because in the event of a crash the data is recovered from the transaction log.
"in the event of a crash the data is recovered from the transaction log"
Doesn't this statement imply that a disk hit occurred before a client is told that a transaction committed (vs. being told that a unique key constraint was violated, etc.)? I'm talking about a more extreme form where I don't have to wait multiple milliseconds for a disk platter to spin around before continuing with my processing.
Re: Redis: new disk storage to replace VM
#17Earlier quoted context omitted.
All modern relational databases implement exactly what you've described... transactions are written to a transaction log, which is flushed to disk every few seconds (or whenever you want to guarantee that a txn is durable). Changes to the actual data need not be persisted in a timely manner, because in the event of a crash the data is recovered from the transaction log.
On MySQL / InnoDB, this is innodb_flush_log_at_trx_commit and how the buffer log is flushed can have a tremendous impact on the latency of writes.
So, no physical disk write need occur before a client can continue with processing? If so, cool.
Re: Redis: new disk storage to replace VM
#18Earlier quoted context omitted.
All modern relational databases implement exactly what you've described... transactions are written to a transaction log, which is flushed to disk every few seconds (or whenever you want to guarantee that a txn is durable). Changes to the actual data need not be persisted in a timely manner, because in the event of a crash the data is recovered from the transaction log.
"in the event of a crash the data is recovered from the transaction log" Doesn't this statement imply that a disk hit occurred before a client is told that a transaction committed (vs. being told that a unique key constraint was violated, etc.)? I'm talking about a more extreme form where I don't have to wait multiple milliseconds for a disk platter to spin around before continuing with my processing.
For full durability, you configure/ask the DB to fsync the transaction log before reporting the transaction committed to the client.
Most people can tolerate a few seconds of data loss, so a sensible config will only fsync every few seconds and will report a transaction committed before it hits the disk. If the DB crashes, you lose those recent transactions in this mode.
All (?) relational databases let you choose which fsync style you want. Most (?) ship with this setting set to the conservative 'fsync on every commit' mode. Once you configure a SQL database with a more relaxed setting you get a database that performs much more similarly to NoSQL. But some people need full durability - or want it for particular transactions. In that mode, you're basically bound by the the number of IOPS your disk can do, but are guaranteed full durability.
Re: Redis: new disk storage to replace VM
#19Earlier quoted context omitted.
Actually, that is talking about the existing virtual-memory (VM) implementation, which swaps data in and out to disk, and doesn't work so great. The change being talked about here is all about replacing that exact flakey VM with a more solid disk backed approach
I'm sorry, but that's what I meant. I'd be more alarmed than enticed to discover that the current implementation of datasets-larger-than-RAM for my chosen database was considered "flakey", and was going to be swapped out for a green-field approach in the next release. For reference, this is the blog post that introduced the VM idea: http://antirez.com/post/redis-virtual-memory-story.html
If Redis was "my chosen database" then I should have done my homework better.
Redis has always been billed as an in-RAM database, which hard-implies that your dataset must fit in RAM, and thus that Redis is unlikely to be your only database.
VM - i.e. even the possibility of having a dataset larger than RAM - is a recent addition, and one flagged as experimental. It had some issues which meant (for us and for the author of the post you linked to above) Redis was still not feasible for datasets larger than RAM. This new diskstore model, and the associated rethink, is very encouraging news, as it means that scenario might one day be possible after all.
Re: Redis: new disk storage to replace VM
#20I am curious as to what people who use Redis in production think of these types of changes. Is this alarming or hopeful? Seems like a rather large shift in trade-offs and a whole new set of tuning parameters to play with.
Based on mailing list traffic and comments from the developers, it seems like most people using Redis in production have datasets that fit in RAM, in which case they will be utterly indifferent to these changes.