41–50 of 82 posts
Calling Postgres experts: Why, exactly, do we need to put a memory cache such as Redis in front of Postgres? Postgres has its own in-memory cache that it updates on reads and writes, right? What makes Postgres' cache so much worse than a dedicated Redis?
It's not worse, this is just a cheap way to increase performance without having to scale the main instance vertically.
Earlier quoted context omitted.
Because you can’t scale out just the cache part of Postgres, one machine can only have so much memory
If you have a second machine, why not just put a Postgres read replica on it? Letting the WAL deal with replica consistency is much simpler than making the client responsible for keeping an external cache in sync, and you get the benefit of keeping everything in Postgres.
Calling Postgres experts: Why, exactly, do we need to put a memory cache such as Redis in front of Postgres? Postgres has its own in-memory cache that it updates on reads and writes, right? What makes Postgres' cache so much worse than a dedicated Redis?
If you have many inserts and deletes on a table, the table will build up tombstones and postgres will eventually be forced to vacuum the table. This doesn't block normal operation, but auto vacuums on large tables can be resource intensive - especially on the storage/io side. And this - at worst - can turn into a resource contention so you either end up with an infinite auto vacuum (because the vacuum can't keep up fast enough), or a severe performance impact on all queries on the system (and since this is your postgres-as-redis, there is a good chance all of the hot paths rely on the cache and get slowed down significantly).
Both of these result in different kinds of fun - either your applications just stop working because postgres is busy cleaning up, or you end up with some horrible table bloat in the future, which will take hours and hours of application downtime to fix, because your drives are fast, but not that fast.
There are ways to work around this, naturally. You could have an expiration key with an index on it, and do "select * from cache order by expiration_key desc limit 1", and throw pg_partman at it to partition the table based on the expiration key, and drop old values by dropping partitions and such... but at some point you start wondering if using a system meant for this kinda workload is easier.
Calling Postgres experts: Why, exactly, do we need to put a memory cache such as Redis in front of Postgres? Postgres has its own in-memory cache that it updates on reads and writes, right? What makes Postgres' cache so much worse than a dedicated Redis?
Sometimes I feel like the articles posted here are elaborate trolls. Like they wanted to know what the purpose of a cache is and how/when to implement one, but they didn't want to do research, so they came up with the worst idea they could imagine and blogged about it, hoping someone on HN would tell them the right way (or just laughing at people trying to correct them). It gives me anxiety how much truly awful advic…
Not super constructive either, the beauty of having things like HN is to be able to discuss ideas, rather than just point to them as wrong. At least a couple of counterpoints on why this would be a bad idea and offer a better approach.
Feedback like root comment is important to keep pushing ideas forward, instead of a loop.
Calling Postgres experts: Why, exactly, do we need to put a memory cache such as Redis in front of Postgres? Postgres has its own in-memory cache that it updates on reads and writes, right? What makes Postgres' cache so much worse than a dedicated Redis?
How would the architecture in the OP mesh with master-slave postgres setups? If I write a cache item how can I be certain the freshest entry is read back from the read-only slave? Can/do I pay a performance penalty on writes waiting for it to be synchronized? Is it better, when it comes to caching, to ignore the slave and send all read/write cache related queries to the master? All of these questions go away or are g…
Similarly to Postgres, Redis replication is also async, which means that replicas can be out-of-sync for a brief period of time.
Earlier quoted context omitted.
How would the architecture in the OP mesh with master-slave postgres setups? If I write a cache item how can I be certain the freshest entry is read back from the read-only slave? Can/do I pay a performance penalty on writes waiting for it to be synchronized? Is it better, when it comes to caching, to ignore the slave and send all read/write cache related queries to the master? All of these questions go away or are g…
They don't really go away, because if you need read-only replicas with PostgreSQL, there is a good chance that you will also need read-only replicas with Redis. Similarly to Postgres, Redis replication is also async, which means that replicas can be out-of-sync for a brief period of time.
This however can come with a lot of issues if you started to use this to ensure consistency across many replicas. Writes are only as fast as the slowest replica, and any hickup on any replica could stall all writes.
What I wasn't sure about - IMO in such a situation, you should rather fix the application to deal with (briefly) stale information, and then you can throw either async postgres replicas at it.. or redis replication, or something based on memcache.
> Benchmarking and comparing UNLOGGED tables against Memcached or Redis is out-of-scope of this article
Calling Postgres experts: Why, exactly, do we need to put a memory cache such as Redis in front of Postgres? Postgres has its own in-memory cache that it updates on reads and writes, right? What makes Postgres' cache so much worse than a dedicated Redis?
If you can keep your entire working set in memory, though, then it probably doesn't matter that much.
Sometimes I feel like the articles posted here are elaborate trolls. Like they wanted to know what the purpose of a cache is and how/when to implement one, but they didn't want to do research, so they came up with the worst idea they could imagine and blogged about it, hoping someone on HN would tell them the right way (or just laughing at people trying to correct them). It gives me anxiety how much truly awful advic…