Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How much faster is Redis at storing a blob of JSON compared to Postgres?

peterbe.com

Post reply on HN
Print view View original

61–70 of 119 posts

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#61

Post by swsieber »

Benjamin_Dobell wrote: post #38
fizx wrote: post #30

Earlier quoted context omitted.

A reasonably configured Postgres is easily sub-millisecond for reads and writes on a small dataset like this. I suspect that the author is doing something quite wrong. People generally reach for Redis too quickly, IMO. Postgres is about as fast, has better tooling, and will slow down gracefully as the dataset increases, rather than evicting records as the limit is hit. That said, I guess if you're relatively junior,…

> That said, I guess if you're relatively junior, it's hard to grossly misconfigure Redis, so that's a point in its favor. Redis persistence (or possibly lack-thereof) is probably quite complicated for a junior developer to understand. I'd actually expect a better outcome with PSQL.

I disagree. But anytime I've been exposed to redis, it's always been as an ephemeral data store.

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#62

Post by burpsnard »

citilife wrote: post #57

I use PostgreSQL regularly and pull (and even search) JSON blobs. We have something like 150 million JSON blobs (dataset is 500Gb) that are searched almost every query, response time is maybe 150-200ms... That's on an AWS t3.medium RDS instance. I don't really know what the hell is going on with this, but something isn't correct.

If its mostly on disk, you're paying ms for block io; get more iops.

If the index used are sensitive to low key diversity, fix that, and maybe shard.

You might prefer a vm type with nvme.

If u can divide the data into more small ones, more indexes and cache fits in ram.

Also, cluster data by what works, time, region, userclass, etc.

Only decompress the blobs at point of use.

Design your app to need only small fast data which cant be cached.

Etc

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#63

Post by peterbe »

Hey, Original blog post author here.

What a lot of people don't understand is that you want it both. The resilience of Postgres and the speed of fetching from Redis.

You have data stored. You want to retrieve it as painlessly (shorter time, less CPU usage, less complexity, etc.) as possible. What are the options? How much faster is Redis? How much RAM is that going to eat up? Different tools have different properties. Run some tests to answer some of those questions. Don't just guess. Sure Redis is going to save some time, but how much? Do you know by heart?

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#64

Post by peterbe »

meritt wrote: post #34
skywhopper wrote: post #9

Alas, there's not enough context or code here to even really know what's going on (or maybe I just don't know Django well enough to guess at what isn't said). It sounds like the author is comparing storing and retrieving a record with a JSON field in a PostgreSQL table versus storing and retrieving a raw string (the raw JSON) in Redis? So it's no surprise that the Redis solution is faster. Besides parsing and indexin…

As he chose to show us Django ORM code, instead of the postgres table definition, I'm guessing he's not exactly a postgres wizard. It reads to me like to me like he's writing to a normalized table (complete with foreign key constraints and likely indexes) and comparing against writing a single serialized blob to to a Redis key. I too would expect the performance improvement to be significantly faster. This guy probab…

I've used Postgres in production since 2002 and speaker that PgOpen. But I'm not an expert. I never will be because I make things with tools. What do you do?

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#65

Post by peterbe »

meritt wrote: post #34
skywhopper wrote: post #9

Alas, there's not enough context or code here to even really know what's going on (or maybe I just don't know Django well enough to guess at what isn't said). It sounds like the author is comparing storing and retrieving a record with a JSON field in a PostgreSQL table versus storing and retrieving a raw string (the raw JSON) in Redis? So it's no surprise that the Redis solution is faster. Besides parsing and indexin…

As he chose to show us Django ORM code, instead of the postgres table definition, I'm guessing he's not exactly a postgres wizard. It reads to me like to me like he's writing to a normalized table (complete with foreign key constraints and likely indexes) and comparing against writing a single serialized blob to to a Redis key. I too would expect the performance improvement to be significantly faster. This guy probab…

"As he chose to show us C++ code, instead of the assembly code, I'm guess he's not exactly a compiler expert."

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#66

Post by [deleted] »

peterbe wrote: post #63

Hey, Original blog post author here. What a lot of people don't understand is that you want it both. The resilience of Postgres and the speed of fetching from Redis. You have data stored. You want to retrieve it as painlessly (shorter time, less CPU usage, less complexity, etc.) as possible. What are the options? How much faster is Redis? How much RAM is that going to eat up? Different tools have different properties…

[deleted]

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#67

Post by ngrilly »

peterbe wrote: post #63

Hey, Original blog post author here. What a lot of people don't understand is that you want it both. The resilience of Postgres and the speed of fetching from Redis. You have data stored. You want to retrieve it as painlessly (shorter time, less CPU usage, less complexity, etc.) as possible. What are the options? How much faster is Redis? How much RAM is that going to eat up? Different tools have different properties…

I'm not sure you are measuring what you think you are measuring. Have you set CONN_MAX_AGE in Django?

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#68

Post by peterbe »

sbuttgereit wrote: post #54
skywhopper wrote: post #9

Alas, there's not enough context or code here to even really know what's going on (or maybe I just don't know Django well enough to guess at what isn't said). It sounds like the author is comparing storing and retrieving a record with a JSON field in a PostgreSQL table versus storing and retrieving a raw string (the raw JSON) in Redis? So it's no surprise that the Redis solution is faster. Besides parsing and indexin…

I found the thinking in the article to be mushy. The title asks a reasonably specific question, but the article then goes on to talk about things in ill-defined ways. It mixes in other layers which aren't directly part of the titular question, but nonetheless influence the answer. In an update, the author goes so far as to state that a new test done differently resulted in a different answer. As you point out... very…

> The article is just anecdote, not really good testing.

Mayhaps. Or, just a snapshot from the trenches of real application development.

We all have to make choices. Different pros and cons. Before jumping into, perhaps run some tests with real stuff and change direction based on that.

In application development is usually doesn't matter what the core difference is between two databases because there are drivers and mappers that you more or less have to use. I.e. the Django ORM makes PostgreSQL convenient.

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#69

Post by magduf »

godot wrote: post #45
beamatronic wrote: post #24

Earlier quoted context omitted.

Couchbase really is the best of both worlds, in one product. Store in memory first (memcached) and disk later, on 0 or more cluster nodes. From the calling API, you can choose whether to block or not for the disk commit. Edit: typos

This is what I found in my experience as well. I founded a startup in 2013 (and ran it for 3 years) and used Couchbase as our primary data store (a saas that integrates with web sites as customers that received tens of millions DAU). With no dev ops person, running it was easy, and we've never had performance or outage issues. I know you can't compare a document store/kv store with a relational DB like PG or Mysql, b…

I can give you one hint. I don't really work with databases, so I'm not well-versed on them, but I can easily name all the major RDBMSes (PG, MySQL, Oracle, etc.), and some of the NoSQL ones. I've done a little work at home playing with Postgres, but that's about it.

I've certainly heard about MongoDB many, many times. I've heard of Redis some. I've heard of CouchDB. But this is the very first time I've ever heard of "Couchbase".

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#70

Post by gonational »

zzzeek wrote: post #49

First off: 1. if you are claiming to compare PostgreSQL to Redis, do NOT test with django or SQLAlchemy or whatever, use the raw DBAPI driver, most likely psycopg2, but you might get much faster results with a more optimized driver like asyncpg. 2. Use the Python profiler to generate profiles and to actually see how much time is spent waiting on the network to get the value.

Thanks for this! I did not know about asyncpg, which is purportedly an average of 3x faster than psycopg2.
Post reply on HN