Earlier quoted context omitted.
How would you cache a write?
You can't cache it, but if you don't care about durability as in this case you can acknowledge it before it is fsync'd.
A write is really fast if you don't do it.
How much faster is Redis at storing a blob of JSON compared to Postgres?
101–110 of 119 posts
Re: How much faster is Redis at storing a blob of JSON compared to Postgres?
#102A few problems: - A comparison of Redis and Postgres for in-memory storage without use of TEMP or UNLOGGED[0] tables? - SQL for table creation and queries not shown - Using an ORM on one side and a more direct client on the other While some data is better than none, the quality of this analysis is in question. [0]: https://www.postgresql.org/docs/current/sql-createtable.html
The redis cache client in Django is effectively an ORM - but a much lighter one. The cache client is also doing compression and uses msgpack for serialization.
I think the comparison, in the context of a regular Django developer, is apt.
Re: How much faster is Redis at storing a blob of JSON compared to Postgres?
#103By default Django closes PostgreSQL connections after handling each web request. I’m not certain but what we’re probably seeing here in latency is mostly the substantial overhead in setting up a PG connection. Redis will still be faster but I’d guess the gap would be significantly tighter with persistent connections enabled.
Really? It doesn't use a connection pool?
https://docs.djangoproject.com/en/2.2/ref/settings/#std:sett... pooling is available, but off by default. I suspect most new projects turn it on, but I have no data to support that.
Re: How much faster is Redis at storing a blob of JSON compared to Postgres?
#104A few problems: - A comparison of Redis and Postgres for in-memory storage without use of TEMP or UNLOGGED[0] tables? - SQL for table creation and queries not shown - Using an ORM on one side and a more direct client on the other While some data is better than none, the quality of this analysis is in question. [0]: https://www.postgresql.org/docs/current/sql-createtable.html
The redis cache client in Django is effectively an ORM - but a much lighter one. The cache client is also doing compression and uses msgpack for serialization. I think the comparison, in the context of a regular Django developer, is apt.
> The redis cache client in Django is effectively an ORM
If you're implying that the redis cache client in Django is comparably stunted, I don't agree. Postgres is a lot more tunable than Redis, and the more complex the underlying machinery the leakier/less powerful the abstraction ORMs can provide without taking on the same complexity. An abstraction layer on top of Redis != an abstraction layer on top of Postgres.
Let me put this in context -- you could literally write a custom Postgres access method[0] (see talk on it[1][2][3]) which did nothing but take data and write it to /dev/null, or put it in a bounded space in memory. Of course, you wouldn't want to do that, but there's no way redis is going to be able to compete with that speed -- and from there you can imagine implementing a subset of Redis which is faster than redis can be (while throwing away a lot of course). This is the kind of complexity an ORM has no business replicating most of the time (I don't know which ORMs support the USING in CREATE TABLE yet), but it's part of what could be out there.
> The cache client is also doing compression and uses msgpack for serialization.
Great for performance but bad for testing -- this skews the results of the analysis, they're not doing the same work.
> I think the comparison, in the context of a regular Django developer, is apt.
I don't disagree, because this is what I'd expect a regular Django/Rails developer to do and call it a day -- but for those who want more serious analysis, this isn't it. It's a great start (good analysis is hard), but people shouldn't be thinking this is a good analysis with reasonable options explored -- UNLOGGED & TEMP tables are pretty low hanging fruit that isn't even mentioned. Initial settings of the Postgres cluster aren't mentioned/tweaked.
This is a pretty naive case and Postgres is near an order of magnitude, while being capable of doing a lot of other things. That's pretty amazing.
[0]: https://www.postgresql.org/docs/12/tableam.html
[1]: https://www.pgcon.org/2019/schedule/events/1374.en.html
[2]: https://anarazel.de/talks/2019-05-30-pgcon-pluggable-table-s...
[3]: https://www.youtube.com/watch?v=5RCkZl5HNiQ&list=PLuJmmKtsV1...
Re: How much faster is Redis at storing a blob of JSON compared to Postgres?
#105So reading from RAM in a single-threaded key/value store with a lightweight text protocol, is faster than reading from disk in a multi-threaded relational database with SQL parsing and ACID semantics. As measured by a heavy web framework running in a slow interpreted language that overshadows the speed differential so much that it's only 16x at the end.
This article is obvious in its conclusion, inaccurate in measurement, and useless for any discussion.
Re: How much faster is Redis at storing a blob of JSON compared to Postgres?
#106Earlier quoted context omitted.
> We've never encountered data loss issues with our read and write heavy Redis services Curious... how does one confirm this? A lost acknowledged write is a very obscure thing and I don't think I would know if we lost writes.
Under normal circumstances Redis doesn't lose writes. If it did, we'd be able to detect it in metrics for cache misses or stale data. Our vector clock state machines wouldn't work. We have very fine monitoring for all of these classes of failure. Operationally, we sequence Redis downtime events. They're very rare. This is when most would be concerned about losing data. We shift traffic to be hitless. Our model is eve…
> We've never encountered data loss issues with our read and write heavy Redis services
> We would tolerate such losses
In the context of the original concern, these are conflicting statements.
The underlying argument here is that you shouldn't use redis for anything you can't tolerate losing. Your use case and architecture is great and I'm glad it works well for you. But at the end of the day, there are workloads that can't tolerate such losses. And for those cases, redis is not a good fit.
Re: How much faster is Redis at storing a blob of JSON compared to Postgres?
#107Earlier quoted context omitted.
I found the thinking in the article to be mushy. The title asks a reasonably specific question, but the article then goes on to talk about things in ill-defined ways. It mixes in other layers which aren't directly part of the titular question, but nonetheless influence the answer. In an update, the author goes so far as to state that a new test done differently resulted in a different answer. As you point out... very…
> The article is just anecdote, not really good testing. Mayhaps. Or, just a snapshot from the trenches of real application development. We all have to make choices. Different pros and cons. Before jumping into, perhaps run some tests with real stuff and change direction based on that. In application development is usually doesn't matter what the core difference is between two databases because there are drivers and…
I'm glad you wrote this article because I think people should put themselves out there more (myself included). By doing so, you open yourself up to feedback, both positive and negative, that can help you improve. You have a good example of this with your 9/29 update.
In the spirit of constructive criticism, I second the "not really good testing" feedback, mostly due to the lack of tuning effort and omission of performance-affecting details like configuration, the wire protocols used, how data is being serialized/deserializaed, etc. (though I am no expert with these technologies so some of this may have been implied). All of that may have been overkill for what you were trying to do, but it's what people look for in benchmarking / "good testing". I hope this helps.
Re: How much faster is Redis at storing a blob of JSON compared to Postgres?
#108Seems a really bad study, since it doesn't mention the exact methodology (caching, exact queries, size of the JSON data, etc.), nor does it attempt to figure out why PostgreSQL is taking 8 ms to do something that should take much less and why reads and writes take the same time, or why Redis is taking 0.5 ms (i.e. millions of CPU cycles) for something that should essentially just do a memcpy.
Re: How much faster is Redis at storing a blob of JSON compared to Postgres?
#109postgresql does TOAST compression of large jsonb blocks. You might be incurring the cost of compression/decompression by postgresql
Uh...TOAST is a system for storing data off page. It doesn't have anything to do with compression? Reading a TOAST actually requires reading two pages, so you're correct that there is read and write overhead. But it isn't CPU bound a la compressing the data.
postgres does TOAST compression of larger jsonb.
I dont have any direct link to the docs, but here's a bug report when such behavior was NOT happening.
https://www.postgresql.org/message-id/20140801182102.2696.87...
if possible, this benchmark should be run again with a largish TOAST_TUPLE_TARGET
Re: How much faster is Redis at storing a blob of JSON compared to Postgres?
#110Earlier quoted context omitted.
Uh...TOAST is a system for storing data off page. It doesn't have anything to do with compression? Reading a TOAST actually requires reading two pages, so you're correct that there is read and write overhead. But it isn't CPU bound a la compressing the data.
postgres does TOAST compression of larger jsonb. I dont have any direct link to the docs, but here's a bug report when such behavior was NOT happening. https://www.postgresql.org/message-id/20140801182102.2696.87... if possible, this benchmark should be run again with a largish TOAST_TUPLE_TARGET