Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How much faster is Redis at storing a blob of JSON compared to Postgres?

peterbe.com

Post reply on HN
Print view View original

51–60 of 119 posts

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#51

Post by rndgermandude »

KaiserPro wrote: post #4

Its faster, but then, thats the point of it. Its like saying that memcache is faster than postgres. they do different things for different purposes. I still wouldn't trust redis for anything other than ephemeral storage. Most of the use cases where we use redis assume that the data will at some point go away. The places where we use postgres assume that data is permanent. Infact, we use both together, and it works re…

Redis can persist and replicate just fine.

Aside from lack of ACID you better hope your dataset never gets too large to fit in memory, tho.

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#53

Post by saberience »

beamatronic wrote: post #24
KaiserPro wrote: post #4

Its faster, but then, thats the point of it. Its like saying that memcache is faster than postgres. they do different things for different purposes. I still wouldn't trust redis for anything other than ephemeral storage. Most of the use cases where we use redis assume that the data will at some point go away. The places where we use postgres assume that data is permanent. Infact, we use both together, and it works re…

Couchbase really is the best of both worlds, in one product. Store in memory first (memcached) and disk later, on 0 or more cluster nodes. From the calling API, you can choose whether to block or not for the disk commit. Edit: typos

Ditto. I loved using Couchbase at my previous job (Blizzard Entertainment) and I'm using at my current gig (FinTech) and I've never understood how such an excellent product has such terrible marketing. It blows Mongo out of the water.

If you're looking for a super low-latency (everything sub-millisecond) document/json store with strong resiliency options, cross-data centre replication, options for optimistic/pessimistic locking, json sub document access and modification, etc etc, I don't know why you'd use anything else.

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#54

Post by sbuttgereit »

skywhopper wrote: post #9

Alas, there's not enough context or code here to even really know what's going on (or maybe I just don't know Django well enough to guess at what isn't said). It sounds like the author is comparing storing and retrieving a record with a JSON field in a PostgreSQL table versus storing and retrieving a raw string (the raw JSON) in Redis? So it's no surprise that the Redis solution is faster. Besides parsing and indexin…

I found the thinking in the article to be mushy.

The title asks a reasonably specific question, but the article then goes on to talk about things in ill-defined ways. It mixes in other layers which aren't directly part of the titular question, but nonetheless influence the answer. In an update, the author goes so far as to state that a new test done differently resulted in a different answer. As you point out... very little relevant detail is included for PostgreSQL (which I know well) and I'm sure insufficient detail for Redis (which I don't know well)... certainly not enough detail and well reasoned testing to merit a public posting on the subject.

That's not to say that the gist of the article is incorrect or unexpected... I would expect a well configured Redis to perform better than PostgreSQL for reading and writing JSON data. But there's nothing I can really learn here that makes clear differences with well understood external influences.

The article is just anecdote, not really good testing.

The article should be titled something along the lines of, "How much faster is my way of using Redis at storing JSON than my way of using PostgreSQL?". Perhaps "Medium" would have been a more appropriate venue as well.... ;-)

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#55

Post by tombert »

saberience wrote: post #53
beamatronic wrote: post #24

Earlier quoted context omitted.

Couchbase really is the best of both worlds, in one product. Store in memory first (memcached) and disk later, on 0 or more cluster nodes. From the calling API, you can choose whether to block or not for the disk commit. Edit: typos

Ditto. I loved using Couchbase at my previous job (Blizzard Entertainment) and I'm using at my current gig (FinTech) and I've never understood how such an excellent product has such terrible marketing. It blows Mongo out of the water. If you're looking for a super low-latency (everything sub-millisecond) document/json store with strong resiliency options, cross-data centre replication, options for optimistic/pessimis…

Bit of a silly question, but how would you say Couchbase compares to something like CouchDB? I've never used Couchbase directly, but I've always really enjoyed the simplicity of CouchDB.

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#57

Post by citilife »

I use PostgreSQL regularly and pull (and even search) JSON blobs. We have something like 150 million JSON blobs (dataset is 500Gb) that are searched almost every query, response time is maybe 150-200ms... That's on an AWS t3.medium RDS instance.

I don't really know what the hell is going on with this, but something isn't correct.

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#58

Post by iliketoworkhard »

throwawayy98121 wrote: post #46
fizx wrote: post #30

Earlier quoted context omitted.

A reasonably configured Postgres is easily sub-millisecond for reads and writes on a small dataset like this. I suspect that the author is doing something quite wrong. People generally reach for Redis too quickly, IMO. Postgres is about as fast, has better tooling, and will slow down gracefully as the dataset increases, rather than evicting records as the limit is hit. That said, I guess if you're relatively junior,…

Isn’t the fundamental difference also use case? You would have to use Postgres if you need actual durability. AFAIK, one outage and your redis cache is gone... unless you’re running an HA cluster (which I dunno if that’s offered by redis). But even that doesn’t mean you have durability.

In general caching systems are volatile and not meant to be used to store data that cannot be afforded to be lost. Of course that doesn't stop clients from storing critical data on there in return for the simple GET/PUT APIs and low latencies.

You can go for more complex caches such as Hazelcast which offers hot restarts by persisting data to disk.

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#59

Post by derefr »

throwawayy98121 wrote: post #46
fizx wrote: post #30

Earlier quoted context omitted.

A reasonably configured Postgres is easily sub-millisecond for reads and writes on a small dataset like this. I suspect that the author is doing something quite wrong. People generally reach for Redis too quickly, IMO. Postgres is about as fast, has better tooling, and will slow down gracefully as the dataset increases, rather than evicting records as the limit is hit. That said, I guess if you're relatively junior,…

Isn’t the fundamental difference also use case? You would have to use Postgres if you need actual durability. AFAIK, one outage and your redis cache is gone... unless you’re running an HA cluster (which I dunno if that’s offered by redis). But even that doesn’t mean you have durability.

Redis can be configured as `appendonly yes; appendfsync always`, which the Redis docs say is “very very slow”, but really would just make Redis work the way Postgres does with its WAL, with probably equivalent performance (or still slightly better, since there’s no checkpointing going on.)

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#60

Post by femto113 »

meritt wrote: post #34
skywhopper wrote: post #9

Alas, there's not enough context or code here to even really know what's going on (or maybe I just don't know Django well enough to guess at what isn't said). It sounds like the author is comparing storing and retrieving a record with a JSON field in a PostgreSQL table versus storing and retrieving a raw string (the raw JSON) in Redis? So it's no surprise that the Redis solution is faster. Besides parsing and indexin…

As he chose to show us Django ORM code, instead of the postgres table definition, I'm guessing he's not exactly a postgres wizard. It reads to me like to me like he's writing to a normalized table (complete with foreign key constraints and likely indexes) and comparing against writing a single serialized blob to to a Redis key. I too would expect the performance improvement to be significantly faster. This guy probab…

I likewise suspect the FK and the index on the created timestamp account for most of the difference.
Post reply on HN