Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How much faster is Redis at storing a blob of JSON compared to Postgres?

peterbe.com

Post reply on HN
Print view View original

41–50 of 119 posts

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#41

Post by krenoten »

xgb84j wrote: post #10
KaiserPro wrote: post #4

Its faster, but then, thats the point of it. Its like saying that memcache is faster than postgres. they do different things for different purposes. I still wouldn't trust redis for anything other than ephemeral storage. Most of the use cases where we use redis assume that the data will at some point go away. The places where we use postgres assume that data is permanent. Infact, we use both together, and it works re…

The article talks about _how much_ it is faster which is the interesting question. For a small project you might not want the added complexity of another database and decide to store JSON blobs in PostgreSQL. In this case it's important to understand the performance trade-off you're making.

Just use a hashmap. Hashmaps in some cases can be like 8 orders of magnitude faster than Redis.

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#42

Post by ngrilly »

Bad benchmark with not enough information to draw any relevant conclusion.

My bet: Connection pooling is not enabled for PostgreSQL (Django default value for CONN_MAX_AGE is 0). A new connection is established for each request, which probably explained most of the overhead. Moreover, the connections are are probably TLS secured by default, which increases the handshake time a lot.

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#43

Post by diminoten »

meritt wrote: post #34
skywhopper wrote: post #9

Alas, there's not enough context or code here to even really know what's going on (or maybe I just don't know Django well enough to guess at what isn't said). It sounds like the author is comparing storing and retrieving a record with a JSON field in a PostgreSQL table versus storing and retrieving a raw string (the raw JSON) in Redis? So it's no surprise that the Redis solution is faster. Besides parsing and indexin…

As he chose to show us Django ORM code, instead of the postgres table definition, I'm guessing he's not exactly a postgres wizard. It reads to me like to me like he's writing to a normalized table (complete with foreign key constraints and likely indexes) and comparing against writing a single serialized blob to to a Redis key. I too would expect the performance improvement to be significantly faster. This guy probab…

Another way to look at this is that he's using Postgres the way most people use Postgres + Django.

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#44

Post by echelon »

KaiserPro wrote: post #4

Its faster, but then, thats the point of it. Its like saying that memcache is faster than postgres. they do different things for different purposes. I still wouldn't trust redis for anything other than ephemeral storage. Most of the use cases where we use redis assume that the data will at some point go away. The places where we use postgres assume that data is permanent. Infact, we use both together, and it works re…

> I still wouldn't trust redis for anything other than ephemeral storage

The Redis AOF (commands flush to disk) and replication story is rock solid.

You can replicate writes to an offline secondary that even has scheduled RDB memory dumps or scheduled AOF rewrites.

We've never encountered data loss issues with our read and write heavy Redis services.

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#45

Post by godot »

beamatronic wrote: post #24
KaiserPro wrote: post #4

Its faster, but then, thats the point of it. Its like saying that memcache is faster than postgres. they do different things for different purposes. I still wouldn't trust redis for anything other than ephemeral storage. Most of the use cases where we use redis assume that the data will at some point go away. The places where we use postgres assume that data is permanent. Infact, we use both together, and it works re…

Couchbase really is the best of both worlds, in one product. Store in memory first (memcached) and disk later, on 0 or more cluster nodes. From the calling API, you can choose whether to block or not for the disk commit. Edit: typos

This is what I found in my experience as well. I founded a startup in 2013 (and ran it for 3 years) and used Couchbase as our primary data store (a saas that integrates with web sites as customers that received tens of millions DAU). With no dev ops person, running it was easy, and we've never had performance or outage issues. I know you can't compare a document store/kv store with a relational DB like PG or Mysql, but I have wondered why couchbase is not as popular as say Mongo.

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#46

Post by throwawayy98121 »

fizx wrote: post #30
skywhopper wrote: post #9

Alas, there's not enough context or code here to even really know what's going on (or maybe I just don't know Django well enough to guess at what isn't said). It sounds like the author is comparing storing and retrieving a record with a JSON field in a PostgreSQL table versus storing and retrieving a raw string (the raw JSON) in Redis? So it's no surprise that the Redis solution is faster. Besides parsing and indexin…

A reasonably configured Postgres is easily sub-millisecond for reads and writes on a small dataset like this. I suspect that the author is doing something quite wrong. People generally reach for Redis too quickly, IMO. Postgres is about as fast, has better tooling, and will slow down gracefully as the dataset increases, rather than evicting records as the limit is hit. That said, I guess if you're relatively junior,…

Isn’t the fundamental difference also use case? You would have to use Postgres if you need actual durability. AFAIK, one outage and your redis cache is gone... unless you’re running an HA cluster (which I dunno if that’s offered by redis). But even that doesn’t mean you have durability.

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#48

Post by airstrike »

cortesoft wrote: post #36
rbranson wrote: post #17

By default Django closes PostgreSQL connections after handling each web request. I’m not certain but what we’re probably seeing here in latency is mostly the substantial overhead in setting up a PG connection. Redis will still be faster but I’d guess the gap would be significantly tighter with persistent connections enabled.

Really? It doesn't use a connection pool?

AFAIK you can pick your poison

https://docs.djangoproject.com/en/2.2/ref/databases/#transac...

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#49

Post by zzzeek »

First off:

1. if you are claiming to compare PostgreSQL to Redis, do NOT test with django or SQLAlchemy or whatever, use the raw DBAPI driver, most likely psycopg2, but you might get much faster results with a more optimized driver like asyncpg.

2. Use the Python profiler to generate profiles and to actually see how much time is spent waiting on the network to get the value.

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#50

Post by dijit »

njharman wrote: post #40
dijit wrote: post #5

“Memory is faster than disk, more at 11”

yeah and in normal operation the OS/PG will have all your data in memory (or all your indexes if very, very much data or under spec'd db server). Disk is for writes.

There is one disk operation that is guaranteed to happen when pushing data into a database like PostgreSQL. It will write data to an append-only binary log.

In Postgres this is called “write-ahead-log” or WAL. On MySQL/MariaDB it is called Binlog.

For a long time the only persistence redis even had was an append only binary log-style system.

Additionally you could tune redis to a maximum persistence threshold which emulated the absolute minimum persistence threshold of a system like Postgres or MySQL.

There is no comparison between these systems. One is designed to treat your data very safely by default and the other is designed for speed.

They both achieve those goals.

I don’t even want to go into the fact that vfs access isn’t still slower than direct memory access (for, I hope, many obvious reasons).

The topic of this thread is the equivalent of judging a fish by its ability to climb a tree.

Post reply on HN