Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How much faster is Redis at storing a blob of JSON compared to Postgres?

peterbe.com

Post reply on HN
Print view View original

111–119 of 119 posts

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#111

Post by alexandercrohde »

KaiserPro wrote: post #4

Its faster, but then, thats the point of it. Its like saying that memcache is faster than postgres. they do different things for different purposes. I still wouldn't trust redis for anything other than ephemeral storage. Most of the use cases where we use redis assume that the data will at some point go away. The places where we use postgres assume that data is permanent. Infact, we use both together, and it works re…

It's actually worse than that. It's apples to oranges because it's asking one service to do X and one service to do Y.

You can configure postgres to behave more like Redis, and it will be faster if you do (at the cost of transactional guarantees).

A more interesting test might have been Postgres configured-like-redis vs Redis.

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#112

Post by tracker1 »

fizx wrote: post #30
skywhopper wrote: post #9

Alas, there's not enough context or code here to even really know what's going on (or maybe I just don't know Django well enough to guess at what isn't said). It sounds like the author is comparing storing and retrieving a record with a JSON field in a PostgreSQL table versus storing and retrieving a raw string (the raw JSON) in Redis? So it's no surprise that the Redis solution is faster. Besides parsing and indexin…

A reasonably configured Postgres is easily sub-millisecond for reads and writes on a small dataset like this. I suspect that the author is doing something quite wrong. People generally reach for Redis too quickly, IMO. Postgres is about as fast, has better tooling, and will slow down gracefully as the dataset increases, rather than evicting records as the limit is hit. That said, I guess if you're relatively junior,…

Probably, hard to tell, the biggest difference, is Redis is likely storing the JSON as a string, and Posgres is storing it as binary json (separated out into distinct values/collections). I'd guess if Postgres was using UTF8 text for the JSON it would be closer. Also, Postgres is probably a bit less efficient on persistance requirements in favor of higher stability and preventing data loss.

In any case, Redis is probably a more natural and better fit (remote API result caching). That said, the comparison is awkward at best. If it were to cache localized results, where the dbms (PostgreSQL) is the source of truth, then adding a caching layer above the db may or may not be worth it.

I think the performance comparisons come down to Granny Smith Apples vs Rome Apples... you can bake a pie with either, but they don't taste the same.

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#113

Post by tracker1 »

peterbe wrote: post #64
meritt wrote: post #34

Earlier quoted context omitted.

As he chose to show us Django ORM code, instead of the postgres table definition, I'm guessing he's not exactly a postgres wizard. It reads to me like to me like he's writing to a normalized table (complete with foreign key constraints and likely indexes) and comparing against writing a single serialized blob to to a Redis key. I too would expect the performance improvement to be significantly faster. This guy probab…

I've used Postgres in production since 2002 and speaker that PgOpen. But I'm not an expert. I never will be because I make things with tools. What do you do?

Personally, I try to understand at least conceptually how the underlying data store connects and serializes the data in question. I'm not a fan of ORM tooling, quite frankly and find it often gets in the way more than helps.

I tend to track versioned changes in scripts, with a config table with a lock and a record of which the current version is, locking and upgrading the db on api service startup. It tends to be very solid, and doesn't rely on potentially unexpected behaviors. At work, a project I'm on uses SQL Server projects which has had a great many issues tracking versions. I've also used several ORM tools across differing platforms and they all have issues. More often than not, I'd rather do things the "hard" way. It's a little more initial effort, but saves in the long run.

Beyond this, without understanding how the data is persisted, it's hard to compare equally. If you used pgsql how you used Redis, I'd be surprised if it was more than 5x as slow. There's also less risk of data loss with PostgreSQL. In this case, data caching foreign api results, Redis is probably the more appropriate tool anyway, but the comparisons in TFA are muddy at best.

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#114

Post by tracker1 »

echelon wrote: post #99
phamilton wrote: post #96

Earlier quoted context omitted.

> We've never encountered data loss issues with our read and write heavy Redis services Curious... how does one confirm this? A lost acknowledged write is a very obscure thing and I don't think I would know if we lost writes.

Under normal circumstances Redis doesn't lose writes. If it did, we'd be able to detect it in metrics for cache misses or stale data. Our vector clock state machines wouldn't work. We have very fine monitoring for all of these classes of failure. Operationally, we sequence Redis downtime events. They're very rare. This is when most would be concerned about losing data. We shift traffic to be hitless. Our model is eve…

https://aphyr.com/posts/283-call-me-maybe-redis

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#115

Post by tracker1 »

beamatronic wrote: post #24
KaiserPro wrote: post #4

Its faster, but then, thats the point of it. Its like saying that memcache is faster than postgres. they do different things for different purposes. I still wouldn't trust redis for anything other than ephemeral storage. Most of the use cases where we use redis assume that the data will at some point go away. The places where we use postgres assume that data is permanent. Infact, we use both together, and it works re…

Couchbase really is the best of both worlds, in one product. Store in memory first (memcached) and disk later, on 0 or more cluster nodes. From the calling API, you can choose whether to block or not for the disk commit. Edit: typos

Would you be willing to rely on this for something like tracking election results?

It really depends on your specific environment in terms of how much you are, or are not willing to lose information and it's all relative. I'm a fan of high performing databases (couchbase), or those more tunable (ScyllaDB/Cassandra), or build for indexing (elasticsearch), or those built for consistency (sql).

Each have their use cases, and there's definitely overlap. Understanding the difference and using the right tool for the job is important.

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#116

Post by tracker1 »

godot wrote: post #45
beamatronic wrote: post #24

Earlier quoted context omitted.

Couchbase really is the best of both worlds, in one product. Store in memory first (memcached) and disk later, on 0 or more cluster nodes. From the calling API, you can choose whether to block or not for the disk commit. Edit: typos

This is what I found in my experience as well. I founded a startup in 2013 (and ran it for 3 years) and used Couchbase as our primary data store (a saas that integrates with web sites as customers that received tens of millions DAU). With no dev ops person, running it was easy, and we've never had performance or outage issues. I know you can't compare a document store/kv store with a relational DB like PG or Mysql, b…

Mongo's marketing team. The near 1:1 logical mapping for Node.js and other languages with object structures. It's effective, and very easy to use.

Compared to a closer example like RethinkDB, which was really cool, but nowhere near the money and marketing team ultimately failed. Rethink started in terms of stable data first, performance tuning from there... mongo went the opposite approach and caught up on the stability later. Of course scaling mongo and administration isn't as easy as a single node, or multiple read-replicas.

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#117

Post by einhverfr »

These sort of benchmarks are both useful and useless, depending on what you want. The architectures of the two are very different and so speed depends very much on what you are trying to do.

Access patterns in Redis are very simple and it is a well optimized single threaded event loop. It is very fast but cannot scale up to multiple cores etc. So in this case, yes, storing and retrieving data will be fast, but uses will be narrower and if you need a high degree of concurrency, Redis will pose some issues for you.

At least in my experience, Redis's key problem is that while it is well optimized it is still a single threaded event loop and therefore with complex use comes a lot of headache. For simple things however, it is really hard to beat.

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#118

Post by bufferoverflow »

jjeaff wrote: post #82
bufferoverflow wrote: post #78

Earlier quoted context omitted.

If it fits in RAM, you might as well put it in memcached. You will likely get a 10X speedup. EDIT: To be fair, RAM is cheap these days, and most real world projects fit into RAM without a crazy budget.

Ram is cheap everywhere but "cloud" instances. Was just noticing the other day, I can get about 128GB of ram on a dedicated instance for the same price I am paying in the cloud for 16GB.

Yeah, cloud prices are insane. I only rent individual servers.

Very few real world projects actually do need the cloud features. It's nice to have, sure, but you're paying a lot for it.

Post reply on HN