Redis: new disk storage to replace VM
groups.google.com
Redis: new disk storage to replace VM
1–10 of 24 posts
Re: Redis: new disk storage to replace VM
#2sounds like he's trying to implement something similar to a 'dbm' (tokyo/kyoto as current/modern implementations) which are k/v caches that seem to intelligently write to disk.
Presumably the keyset can still be in ram (so that [star]foo[star]bar[star]1 searches on keys can work?), but the dbm model is a fairly efficient key/val implementation, and tokyo/kyoto is fast, and fairly smart about writing to disk, although I haven't explicitly testing their limitations as you approach ram limits in production.
Not sure what tradeoffs are in mind, but atleast a feature/perf comparison to kyoto compared with diskstore as an internal back for redis would be interesting
[1](doesnt seem i can escape an *)
Re: Redis: new disk storage to replace VM
#3So, as a non-Redis user, am I correct in my understanding that an admin may define a maximum delay before write-behind persistence must be attempted? For applications which can afford to lose a few seconds of data in the case of a failure, this seems like a great way to improve latency.
More generally, why are there no MySQL (or whatever) engines which offer similar capabilities? Wouldn't it be possible for DB clients to "commit" transactions to memory and then have them flushed to disk asynchronously with all other ACI(d) properties maintained? There are many applications which can survive a few seconds of data loss but need transactional properties to avoid data corruption.
Re: Redis: new disk storage to replace VM
#4So, as a non-Redis user, am I correct in my understanding that an admin may define a maximum delay before write-behind persistence must be attempted? For applications which can afford to lose a few seconds of data in the case of a failure, this seems like a great way to improve latency. More generally, why are there no MySQL (or whatever) engines which offer similar capabilities? Wouldn't it be possible for DB client…
This is kinda what mongodb[0] does now and is a question of durability[1]. Current rdbms systems that lay claim to consistency must flush to disk to ensure ACID[2] compliance. Redis does not claim such compliance and I am of the opinion that it should not.
[1]http://en.wikipedia.org/wiki/Durability_(database_systems)
Re: Redis: new disk storage to replace VM
#5So, as a non-Redis user, am I correct in my understanding that an admin may define a maximum delay before write-behind persistence must be attempted? For applications which can afford to lose a few seconds of data in the case of a failure, this seems like a great way to improve latency. More generally, why are there no MySQL (or whatever) engines which offer similar capabilities? Wouldn't it be possible for DB client…
All modern relational databases implement exactly what you've described... transactions are written to a transaction log, which is flushed to disk every few seconds (or whenever you want to guarantee that a txn is durable). Changes to the actual data need not be persisted in a timely manner, because in the event of a crash the data is recovered from the transaction log.
Re: Redis: new disk storage to replace VM
#6Re: Redis: new disk storage to replace VM
#7So, as a non-Redis user, am I correct in my understanding that an admin may define a maximum delay before write-behind persistence must be attempted? For applications which can afford to lose a few seconds of data in the case of a failure, this seems like a great way to improve latency. More generally, why are there no MySQL (or whatever) engines which offer similar capabilities? Wouldn't it be possible for DB client…
All modern relational databases implement exactly what you've described... transactions are written to a transaction log, which is flushed to disk every few seconds (or whenever you want to guarantee that a txn is durable). Changes to the actual data need not be persisted in a timely manner, because in the event of a crash the data is recovered from the transaction log.
On MySQL / InnoDB, this is innodb_flush_log_at_trx_commit and how the buffer log is flushed can have a tremendous impact on the latency of writes.
Re: Redis: new disk storage to replace VM
#8sounds like he's trying to implement something similar to a 'dbm' (tokyo/kyoto as current/modern implementations) which are k/v caches that seem to intelligently write to disk. Presumably the keyset can still be in ram (so that [star]foo[star]bar[star]1 searches on keys can work?), but the dbm model is a fairly efficient key/val implementation, and tokyo/kyoto is fast, and fairly smart about writing to disk, although…
I don't think the keyset will be in ram: "Redis will never use more RAM, even if we have 2 MB of max memory and 1 billion of keys. This works since now we don't need to take keys in memory."
I haven't looked at the current code to see if there is a way to favor keeping the keys in memory, but it would seem that wildcard searches here can/will be disk-bound.
Re: Redis: new disk storage to replace VM
#9sounds like he's trying to implement something similar to a 'dbm' (tokyo/kyoto as current/modern implementations) which are k/v caches that seem to intelligently write to disk. Presumably the keyset can still be in ram (so that [star]foo[star]bar[star]1 searches on keys can work?), but the dbm model is a fairly efficient key/val implementation, and tokyo/kyoto is fast, and fairly smart about writing to disk, although…
I don't think the keyset will be in ram: "Redis will never use more RAM, even if we have 2 MB of max memory and 1 billion of keys. This works since now we don't need to take keys in memory." I haven't looked at the current code to see if there is a way to favor keeping the keys in memory, but it would seem that wildcard searches here can/will be disk-bound.
tokyo supports a b-tree index on the keys written to disk which would optimize blah* queries but not [wildcard]foo[wildcard], and then writes become log(N)
Re: Redis: new disk storage to replace VM
#10So, as a non-Redis user, am I correct in my understanding that an admin may define a maximum delay before write-behind persistence must be attempted? For applications which can afford to lose a few seconds of data in the case of a failure, this seems like a great way to improve latency. More generally, why are there no MySQL (or whatever) engines which offer similar capabilities? Wouldn't it be possible for DB client…
Yeah, basically what you do is configure how often you want Redis to sync to the disk. You define a period of time and a number of records, and if that amount of time passes and that number of records have been changed, it saves to the disk.
This is the default:
save 900 1
save 300 10
save 60 10000