Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How much faster is Redis at storing a blob of JSON compared to Postgres?

peterbe.com

Post reply on HN
Print view View original

71–80 of 119 posts

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#71

Post by threeseed »

fizx wrote: post #30
skywhopper wrote: post #9

Alas, there's not enough context or code here to even really know what's going on (or maybe I just don't know Django well enough to guess at what isn't said). It sounds like the author is comparing storing and retrieving a record with a JSON field in a PostgreSQL table versus storing and retrieving a raw string (the raw JSON) in Redis? So it's no surprise that the Redis solution is faster. Besides parsing and indexin…

A reasonably configured Postgres is easily sub-millisecond for reads and writes on a small dataset like this. I suspect that the author is doing something quite wrong. People generally reach for Redis too quickly, IMO. Postgres is about as fast, has better tooling, and will slow down gracefully as the dataset increases, rather than evicting records as the limit is hit. That said, I guess if you're relatively junior,…

You have no evidence that:

a) PostgreSQL has sub-millisecond read/writes for storing JSON blobs than Redis.

b) Java results in buggier code than Javascript.

If you're going to make bold claims the onus is on you to provide evidence to back it up.

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#72

Post by radiospiel »

I saw something similar recently when reading JSON data from postgres via Rails, which was unbelievably slow. I found the reason to be that Rails would automatically parse the JSON value into a in-memory ruby object. Since this was for a API-like backend, the solution was simple: just read the column as a string ("SELECT payload::varchar..")

I don't know if dkango is doing the same, but using an ORM might just do that to you.

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#73

Post by threeseed »

Benjamin_Dobell wrote: post #38
fizx wrote: post #30

Earlier quoted context omitted.

A reasonably configured Postgres is easily sub-millisecond for reads and writes on a small dataset like this. I suspect that the author is doing something quite wrong. People generally reach for Redis too quickly, IMO. Postgres is about as fast, has better tooling, and will slow down gracefully as the dataset increases, rather than evicting records as the limit is hit. That said, I guess if you're relatively junior,…

> That said, I guess if you're relatively junior, it's hard to grossly misconfigure Redis, so that's a point in its favor. Redis persistence (or possibly lack-thereof) is probably quite complicated for a junior developer to understand. I'd actually expect a better outcome with PSQL.

Not sure what you are getting at here.

Redis is one of the simplest data stores around to use and its persistence is solid.

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#74

Post by threeseed »

godot wrote: post #45
beamatronic wrote: post #24

Earlier quoted context omitted.

Couchbase really is the best of both worlds, in one product. Store in memory first (memcached) and disk later, on 0 or more cluster nodes. From the calling API, you can choose whether to block or not for the disk commit. Edit: typos

This is what I found in my experience as well. I founded a startup in 2013 (and ran it for 3 years) and used Couchbase as our primary data store (a saas that integrates with web sites as customers that received tens of millions DAU). With no dev ops person, running it was easy, and we've never had performance or outage issues. I know you can't compare a document store/kv store with a relational DB like PG or Mysql, b…

> but I have wondered why couchbase is not as popular as say Mongo

Because MongoDB is a single binary to download and install and has one of the simplest config files around.

It's arguably the easiest database to use and operate.

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#75

Post by threeseed »

volkk wrote: post #32
dsr_ wrote: post #20

Earlier quoted context omitted.

Sorry, no. The complexity is that you are now relying on another server which can break in new ways, requires startup and shutdown and configuration and possibly load-balancing; management of all the above... updates, security reviews, and documentation. Will you need new server types with an emphasis on RAM? How many and when? How are you monitoring it for errors and performance? That's the difference between thinki…

True, but a lot of that can be solved with PaaS/DBaaS products like compose or something

Shame you got downvoted because it's a valid point.

These days most databases are available as a managed service and Redis in particular is a standard feature on AWS, Azure and Google Cloud.

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#76

Post by unnouinceput »

Quote:"My current implementation uses PostgreSQL (via the Django ORM) to store this stuff."

At ORM the credibility of the author was all lost on me. I mean you went from cloud to local cache as a performance improvement and then you chose to do it via ORM? Why not make your own custom solution if you went this road anyway? It's like changing your alarms in your house from using the ones provided by a security company with making your own because you're an electronics engineer and then you build them using electronic kits from commerce and complain that using a different kit is better. It's your house dude, make them all custom for maximum efficiency if you went this way.

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#77

Post by jdub »

peterbe wrote: post #63

Hey, Original blog post author here. What a lot of people don't understand is that you want it both. The resilience of Postgres and the speed of fetching from Redis. You have data stored. You want to retrieve it as painlessly (shorter time, less CPU usage, less complexity, etc.) as possible. What are the options? How much faster is Redis? How much RAM is that going to eat up? Different tools have different properties…

"Do you know by heart?" is very much the wrong question though. :-)

Benchmarking is surprisingly hard. After some digging, you might discover that you're actually measuring how long it takes to bring up a TCP connection, or how long it takes PostgreSQL to serialise the JSON from its stored form. It could be differences in the driver layer Django is using. Coming up with the corrct experiment can be the toughest part of doing the science.

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#78

Post by bufferoverflow »

mslot wrote: post #23
ninth_ant wrote: post #11

Something is very wrong for the median read of what should be a moderate sized blob with an indexed lookup to be 8ms. The article doesn’t give enough details to really understand what is that is happening here, but I would recommend everyone to take this 14/16x report and not expect that to be the general case.

Looking up a row of JSONB data by a primary key in a 3GB PostgreSQL table on my laptop takes 0.4-0.5ms, since it easily fits in memory and Postgres caches the data.

If it fits in RAM, you might as well put it in memcached. You will likely get a 10X speedup.

EDIT:

To be fair, RAM is cheap these days, and most real world projects fit into RAM without a crazy budget.

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#79

Post by Someone »

fizx wrote: post #30
skywhopper wrote: post #9

Alas, there's not enough context or code here to even really know what's going on (or maybe I just don't know Django well enough to guess at what isn't said). It sounds like the author is comparing storing and retrieving a record with a JSON field in a PostgreSQL table versus storing and retrieving a raw string (the raw JSON) in Redis? So it's no surprise that the Redis solution is faster. Besides parsing and indexin…

A reasonably configured Postgres is easily sub-millisecond for reads and writes on a small dataset like this. I suspect that the author is doing something quite wrong. People generally reach for Redis too quickly, IMO. Postgres is about as fast, has better tooling, and will slow down gracefully as the dataset increases, rather than evicting records as the limit is hit. That said, I guess if you're relatively junior,…

There’s

  class AmazonAffiliateLookup(models.Model, TotalCountMixin)
I guess that TotalCountMixin adds some triggers that maintain a count of the records in the table in a separate table (could or could not be a good idea, depending on the table size, relative frequency of ‘give me the number of items’ calls relative to inserts and deletes, etc, but might introduce a bottleneck on updating that count)

There also is:

  song = models.ForeignKey(Song, on_delete=models.CASCADE)
I don’t understand why that foreign key is there, but it can’t speed up inserts and deletes.

Then, we have:

  created = models.DateTimeField(auto_now_add=True, db_index=True)
I think creating an index is done to implement the equivalent of Redis’ auto-expiry.

Again, that may be suboptimal if there are few records in the database. DateTimeField also may be ‘too structured’ for the task at hand (it might needlessly parse year/month/day out of time stamp in nanoseconds, for example, or even do time zone calculations to convert a local time to UTC), but I don’t know. It would be helpful if the text showed the actual SQL DDL.

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#80

Post by lstamour »

magduf wrote: post #69
godot wrote: post #45

Earlier quoted context omitted.

This is what I found in my experience as well. I founded a startup in 2013 (and ran it for 3 years) and used Couchbase as our primary data store (a saas that integrates with web sites as customers that received tens of millions DAU). With no dev ops person, running it was easy, and we've never had performance or outage issues. I know you can't compare a document store/kv store with a relational DB like PG or Mysql, b…

I can give you one hint. I don't really work with databases, so I'm not well-versed on them, but I can easily name all the major RDBMSes (PG, MySQL, Oracle, etc.), and some of the NoSQL ones. I've done a little work at home playing with Postgres, but that's about it. I've certainly heard about MongoDB many, many times. I've heard of Redis some. I've heard of CouchDB. But this is the very first time I've ever heard of…

It was popular briefly on HN back in 2015, so it had a late start compared to Mongo which was maybe 2012? And the commercialization of memcached was perhaps noticed most by PHP devs, as memcached is common enough with Wordpress and Drupal, though Redis might have overtaken it some. I always thought couchbase was a very interesting idea but never had the chance to seriously suggest it at work. These endorsements actually help me consider it, but at the time it seemed like they didn’t do a great job explaining who was using it and why/how it made a difference. I’ve heard more, recently, about using DocumentDB or CockroachDB than it, actually. And most places I work at are still riding the Postgres/Oracle/SQL traditional database shop, where it’s easier to introduce Redis or Memcached than to switch where all the data is stored. Using it for a new project would require buy in, but might be viable.
Post reply on HN