Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Redis is fast – I'll cache in Postgres

dizzy.zone

Post reply on HN
Print view View original

31–40 of 311 posts

Re: Redis is fast – I'll cache in Postgres

#31

Post by maxbond »

frollogaston wrote: post #27
cortesoft wrote: post #23

Having the ability to set a TTL on the cache key is a critical feature of a cache, not something that can be tacked on later. I always find these "don't use redis" posts kind of strange. Redis is so simple to operate at any scale, I don't quite get why it is important to remove it.

Yeah, the article was like "I always need a DB anyway" but then sets up an extra cronjob to expire keys, plus more code. I get YAGNI and avoiding deps, but this is really extra stuff to deal with. Maybe Postgres could use a caching feature. Until then, I'm gonna drop in Redis or memcached instead of reinventing the wheel.

Expiring keys in Postgres with a created_at column and a pg_cron job is very easy (at least, if you're comfortable in Postgres). Redis is world class though of course, and can be deployed turn-key in basically any environment. If you're more comfortable in Redis than Postgres, more power to you. Different choices can be pragmatic to different people.

Personally for a greenfield project, my thinking would be that I am paying for Postgres already. So I would want to avoid paying for Redis too. My Postgres database is likely to be underutilized until (and unless) I get any real scale. So adding caching to it is free in terms of dollars.

Re: Redis is fast – I'll cache in Postgres

#32

Post by not_kurt_godel »

maxbond wrote: post #26
MobiusHorizons wrote: post #16

Earlier quoted context omitted.

“Premature optimization” typically refers to optimizing before profiling. Ie optimizing in places that won’t help. Is redis not improving your latency? Is it adding complexity that isn’t worth it? Why bother removing it?

I like to call cases like this "premature distribution." Or maybe you could call it "premature capacity." If you have an application running in the cloud with several thousand requests per day, you could probably really benefit from adding a service like Redis. But when you have 0-10 users and 0-1000 requests per day, it can make more sense to write something more monolithic and with limited scalability. Eg, doing ev…

You probably don't need Redis until you have thousands of requests per minute, nevermind per day.

Re: Redis is fast – I'll cache in Postgres

#33

Post by frollogaston »

maxbond wrote: post #31
frollogaston wrote: post #27

Earlier quoted context omitted.

Yeah, the article was like "I always need a DB anyway" but then sets up an extra cronjob to expire keys, plus more code. I get YAGNI and avoiding deps, but this is really extra stuff to deal with. Maybe Postgres could use a caching feature. Until then, I'm gonna drop in Redis or memcached instead of reinventing the wheel.

Expiring keys in Postgres with a created_at column and a pg_cron job is very easy (at least, if you're comfortable in Postgres). Redis is world class though of course, and can be deployed turn-key in basically any environment. If you're more comfortable in Redis than Postgres, more power to you. Different choices can be pragmatic to different people. Personally for a greenfield project, my thinking would be that I am…

I'm comfy with Postgres though, like I'll center my entire backend around it and do the heavy lifting in SQL (never ORM). It's more that I don't want to depend on a cronjob for something as fundamental as a cache.

Usually Postgres costs a lot more than Redis if you're paying for a platform. Like a decent Redis or memcached in Heroku is free. And I don't want to waste precious Postgres connections or risk bogging down the whole DB if there's lots of cache usage, which actually happened last time I tried skipping Redis.

Re: Redis is fast – I'll cache in Postgres

#34

Post by nvader »

I hope you remember to add monitoring for your cache expiry cron job so you can be notified if it ever fails to run due to a code change or configuration change. For example, database credentials might be rolled and if you forget to update the job your primary database could fill up.

Perhaps you could have a second cron job that runs to verify that the first one completed. It could look for a last-ran entry. You should put it in the same database, so maybe perhaps you could a key value store like redis for that.

Re: Redis is fast – I'll cache in Postgres

#36

Post by coder543 »

cortesoft wrote: post #23

Having the ability to set a TTL on the cache key is a critical feature of a cache, not something that can be tacked on later. I always find these "don't use redis" posts kind of strange. Redis is so simple to operate at any scale, I don't quite get why it is important to remove it.

I keep hoping Postgres will one day have the ability to mark a timestamp column as an expiry column. It would be useful for all kinds of things beyond caching, including session tokens, feature flags, background jobs, rate limiting, delayed deletion (as a variant of soft deletion), etc.

It seems like the autovacuum could take care of these expired rows during its periodic vacuum. The query planner could automatically add a condition that excludes any expired rows, preventing expired rows from being visible before autovacuum cleans them up.

Re: Redis is fast – I'll cache in Postgres

#37

Post by monkeyelite »

nvader wrote: post #34

I hope you remember to add monitoring for your cache expiry cron job so you can be notified if it ever fails to run due to a code change or configuration change. For example, database credentials might be rolled and if you forget to update the job your primary database could fill up. Perhaps you could have a second cron job that runs to verify that the first one completed. It could look for a last-ran entry. You shou…

This is more than a few leaps of logic. a bad config of any infrastructure can cause problems

Re: Redis is fast – I'll cache in Postgres

#40

Post by hnaccountme »

I think there is more performance to be taken from PostgreSQL. Not sure how pgx works internally since I have not used it.

There are async functions provided by PostgreSQL client library (libpq). I've used it to process around 2000 queries on a single connection per second on a logged table.

Post reply on HN