Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How much faster is Redis at storing a blob of JSON compared to Postgres?

peterbe.com

Post reply on HN
Print view View original

91–100 of 119 posts

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#91

Post by erichocean »

threeseed wrote: post #73
Benjamin_Dobell wrote: post #38

Earlier quoted context omitted.

> That said, I guess if you're relatively junior, it's hard to grossly misconfigure Redis, so that's a point in its favor. Redis persistence (or possibly lack-thereof) is probably quite complicated for a junior developer to understand. I'd actually expect a better outcome with PSQL.

Not sure what you are getting at here. Redis is one of the simplest data stores around to use and its persistence is solid.

> Redis is one of the simplest data stores around to use and its persistence is solid.

I've lost countless amount of data in production systems running Redis. Haven't ever lost anything in Postgres. (I still run both.)

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#92

Post by foobarian »

jefurii wrote: post #18
kieselguhr_kid wrote: post #8

Earlier quoted context omitted.

One of the shops I worked in used Redis as the primary datastore. While there was a lot of extra complexity around relational data (especially referential integrity), the snapshotting systems worked very well and we didn't have to worry about data loss. I don't necessarily think this was an ideal setup (I can't tell you how many headaches Postgres would have solved for us), but I think there are plenty of cases where…

> but I think there are plenty of cases where you could use it without fear that your data will evaporate. What about "redis-cli flushall"? (Don't type this on your production cluster).

Protip: don't use flushall in production, use flushall async or you will have a bad time

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#93

Post by Justsignedup »

The problem of redis isn't speed of single object storage and retrieval.

We know for a fact that redis stores shit fast. And a blob of json is perfect.

The problem is multi-fold. Now you have a schema-on-read. You may spend a lot more processing time querying and accessing the data rather than getting exactly what you want from a sql query. Concurrency and locking. Constraints dbs offer.

So it is important to evaluate on more than JUST read/write speed.

I worked on a team that used nosql for their db needs. The performance was so terribly slow, not because of read/write speeds, but because of everything I listed above. They had to basically manually write a query optimizer vs sql. They quickly swapped to postgres and had somewhere along the lines of 10-100x performance (depending on what parts) improvements with only a week of work.

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#94

Post by bmm6o »

peterbe wrote: post #64
meritt wrote: post #34

Earlier quoted context omitted.

As he chose to show us Django ORM code, instead of the postgres table definition, I'm guessing he's not exactly a postgres wizard. It reads to me like to me like he's writing to a normalized table (complete with foreign key constraints and likely indexes) and comparing against writing a single serialized blob to to a Redis key. I too would expect the performance improvement to be significantly faster. This guy probab…

I've used Postgres in production since 2002 and speaker that PgOpen. But I'm not an expert. I never will be because I make things with tools. What do you do?

Try not to take it personally. With relevant details not made clear, people have to try to fill in the blanks. Wondering about the table definition is definitely understandable.

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#95

Post by hardwaresofton »

A few problems:

- A comparison of Redis and Postgres for in-memory storage without use of TEMP or UNLOGGED[0] tables?

- SQL for table creation and queries not shown

- Using an ORM on one side and a more direct client on the other

While some data is better than none, the quality of this analysis is in question.

[0]: https://www.postgresql.org/docs/current/sql-createtable.html

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#96

Post by phamilton »

echelon wrote: post #44
KaiserPro wrote: post #4

Its faster, but then, thats the point of it. Its like saying that memcache is faster than postgres. they do different things for different purposes. I still wouldn't trust redis for anything other than ephemeral storage. Most of the use cases where we use redis assume that the data will at some point go away. The places where we use postgres assume that data is permanent. Infact, we use both together, and it works re…

> I still wouldn't trust redis for anything other than ephemeral storage The Redis AOF (commands flush to disk) and replication story is rock solid. You can replicate writes to an offline secondary that even has scheduled RDB memory dumps or scheduled AOF rewrites. We've never encountered data loss issues with our read and write heavy Redis services.

> We've never encountered data loss issues with our read and write heavy Redis services

Curious... how does one confirm this? A lost acknowledged write is a very obscure thing and I don't think I would know if we lost writes.

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#97

Post by phamilton »

krenoten wrote: post #41
xgb84j wrote: post #10

Earlier quoted context omitted.

The article talks about _how much_ it is faster which is the interesting question. For a small project you might not want the added complexity of another database and decide to store JSON blobs in PostgreSQL. In this case it's important to understand the performance trade-off you're making.

Just use a hashmap. Hashmaps in some cases can be like 8 orders of magnitude faster than Redis.

Hashmaps are faster, except when you need the data to be available to multiple machines.

A good description of redis is a "shared heap". HashSets, HashMaps, Heaps, Priority Queues, etc. are all great and fast in an application, but once you need to start sharing that data things get complicated quickly. So you designate a single server to implement those data structures and expose them to your application. And what you end up with is basically redis.

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#98

Post by [deleted] »

jjeaff wrote: post #83
peterbe wrote: post #68

Earlier quoted context omitted.

> The article is just anecdote, not really good testing. Mayhaps. Or, just a snapshot from the trenches of real application development. We all have to make choices. Different pros and cons. Before jumping into, perhaps run some tests with real stuff and change direction based on that. In application development is usually doesn't matter what the core difference is between two databases because there are drivers and…

Normally, I would agree that it's a snapshot or data point. But one of the things that I don't see mentioned anywhere else is that this is running on a macbook. I don't know how it works with his setup, but I get significantly different performance results on my local dev system (windows) than a linux server. I run everything in hyper-v docker, which should actually get pretty close to bare-metal performance (as hype…

[deleted]

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#99

Post by echelon »

phamilton wrote: post #96
echelon wrote: post #44

Earlier quoted context omitted.

> I still wouldn't trust redis for anything other than ephemeral storage The Redis AOF (commands flush to disk) and replication story is rock solid. You can replicate writes to an offline secondary that even has scheduled RDB memory dumps or scheduled AOF rewrites. We've never encountered data loss issues with our read and write heavy Redis services.

> We've never encountered data loss issues with our read and write heavy Redis services Curious... how does one confirm this? A lost acknowledged write is a very obscure thing and I don't think I would know if we lost writes.

Under normal circumstances Redis doesn't lose writes. If it did, we'd be able to detect it in metrics for cache misses or stale data. Our vector clock state machines wouldn't work. We have very fine monitoring for all of these classes of failure.

Operationally, we sequence Redis downtime events. They're very rare. This is when most would be concerned about losing data.

We shift traffic to be hitless. Our model is eventually consistent across multiple regions. We won't accept writes in a cluster that is going down.

The Redis replication chain was chosen so that the primary Redis doesn't have to fork (aofrewrite, bgsave) or even do disk IO (AOF ops in general). We let our offline replicas do that work. The primary Redis can focus on serving reads and writes.

We alert on replication chain breakdown. We sequence traffic draining and primary / secondary swap operations for things like OS or Redis upgrades.

It's pretty sophisticated and largely automated.

This model tolerates losses of Redis instances. Only writes to the primary occurring in the short time before replication while not failed out might be lost. But that number would be incredibly, incredibly small. We would tolerate such losses.

We've got a lot of nines of reliability, and this pattern has scaled and served us well.

Re: How much faster is Redis at storing a blob of JSON compared to Postgres?

#100

Post by doliveira »

peterbe wrote: post #68
sbuttgereit wrote: post #54

Earlier quoted context omitted.

I found the thinking in the article to be mushy. The title asks a reasonably specific question, but the article then goes on to talk about things in ill-defined ways. It mixes in other layers which aren't directly part of the titular question, but nonetheless influence the answer. In an update, the author goes so far as to state that a new test done differently resulted in a different answer. As you point out... very…

> The article is just anecdote, not really good testing. Mayhaps. Or, just a snapshot from the trenches of real application development. We all have to make choices. Different pros and cons. Before jumping into, perhaps run some tests with real stuff and change direction based on that. In application development is usually doesn't matter what the core difference is between two databases because there are drivers and…

Then cat your data to /dev/null because it will blow both Redis and PostgreSQL out of the water when it comes to benchmarks.

Point being they serve very different purposes, it's not about benchmarks.

Post reply on HN