Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Fast can A Single Instance of Redis be?

docs.keydb.dev

Post reply on HN
Print view View original

51–60 of 79 posts

Re: How Fast can A Single Instance of Redis be?

#51

Post by KenanSulayman »

segmondy wrote: post #35
PopeDotNinja wrote: post #33

Earlier quoted context omitted.

Depends on how you define database. Wikipedia starts with... ”A database is an organized collection of data, generally stored and accessed electronically from a computer system. Where databases are more complex they are often developed using formal design and modeling techniques." [1] ...which makes Redis sound like a database to me. If I am not technically correct, feel to educate me. Side note: I try to focus on wh…

Your filesystem then is a database. You can even stretch it to include your text editor. Redis is not a database.

Of course the filesystem is a database. In fact, bank mainframes still use distributed file systems as “databases” where file writes are auditable and fully reversible transactions. (I worked on such systems.)

Re: How Fast can A Single Instance of Redis be?

#52

Post by privateSFacct »

dvirsky wrote: post #15

IIUC the benchmark did not use pipelines and the numbers show that. Redis itself can outperform this by a factor of 10 with pipelining, and I'm betting the difference from the module won't be that big given such a setup (which is how you should ideally work anyway). Still, nice to see the modules API being put to interesting use cases.

Yeah - be interesting to see the real use case here. Pipelining + scripting if needed normally go a long way. I'm curious who needs even more beyond this in terms of performance and if this module actually provides it.

Re: How Fast can A Single Instance of Redis be?

#53

Post by dvirsky »

KenanSulayman wrote: post #48
segmondy wrote: post #27

"Redis is known as one of the fastest databases out there." Redis is not a database. Let's begin with that.

I very much beg to differ. Just because a database maps to memory and doesn’t commit to disk per transaction doesn’t make it less of a database. So if not a database, what is it?

Also, redis does commit to disk per transaction if you want it to. IMO it should start doing it by default.

Re: How Fast can A Single Instance of Redis be?

#54

Post by jdsully »

lossolo wrote: post #12

Well, to be honest if you are IO/network bound and kernel TCP stack is the bottleneck then user space networking like DPDK can help in every application. It depends on application but sometimes additional complexity of introducing DPDK is just not worth it, and just spinning another instance/server is a better choice. Look also at Seastar used in ScyllaDB with and without DPDK numbers. Just remember you need to give…

DPDK does poll but you can have it sleep in low traffic situations. At the expense of a little extra latency.

Re: How Fast can A Single Instance of Redis be?

#55

Post by im3w1l »

segmondy wrote: post #35
PopeDotNinja wrote: post #33

Earlier quoted context omitted.

Depends on how you define database. Wikipedia starts with... ”A database is an organized collection of data, generally stored and accessed electronically from a computer system. Where databases are more complex they are often developed using formal design and modeling techniques." [1] ...which makes Redis sound like a database to me. If I am not technically correct, feel to educate me. Side note: I try to focus on wh…

Your filesystem then is a database. You can even stretch it to include your text editor. Redis is not a database.

I think there is much to be learned by comparing databases and filesystems.

One thing I think that filesystems can learn from databases is the notion of a compound primary key. It would be neat if app-files were identified by an (app, type, id) tuple. This would bring the advantages of both the posix and the windows filesystem layouts.

For instance if we had(app=firefox, type=/usr/bin, id=main). Then we could easily find all firefox files by querying by app. Or we could easily find all binaries in PATH by querying by type.

ps. I think this would work better than the overly general tag-based filesystem people sometimes propose.

Re: How Fast can A Single Instance of Redis be?

#56

Post by tnolet »

overcast wrote: post #24
ericblenkarn wrote: post #17

Earlier quoted context omitted.

We are back up, had to move to a bigger server haha

I'm always curious what kind of setup people run that can not handle a thousand people connecting into HTTP.

I had a couple of posts go first page on HN in the last months. I run my blog on Ghost (the hosted SaaS version) and it's crazy fast. I have no idea how they do it, alas.

Re: How Fast can A Single Instance of Redis be?

#57

Post by neomantra »

tinktank wrote: post #5

How is this different from what Solarflare ( https://twitter.com/Solarflare_Comm/status/11134717313798430... ) is doing with Cloud Onload? From what I understand they don't require any application changes and they work on any networking application.

OpenOnload transparently replaces the UDP/TCP network stack and epoll calls of an applications with highly tunable userspace components. It can work with any application that uses these system calls. Of course, that application could be Redis. If you search around, I've commented on using them together.

This is a Redis module, so will only work with Redis. Although I don't see its implementation (?), it appears to connect Redis' Unix Socket interface to a network stack running on DPDK (a user-space low-level network interface).

In SolarFlare world, ef_vi is their library to DPDK -- it is a packet buffer interface. They then have OpenOnload (transparent acceleration that cooperates well with the kernel) and TCPDirect, a proprietary userspace TCP/UDP library that has its own interface. It's even higher performance than OpenOnload because it doesn't have to coordinate with the kernel and you manage the sharing of the network resources.

SolarFlare has a DPDK driver too. OpenOnload doesn't accelerate UNIX sockets.

One thing this module doesn't accelerate is epoll... I think a properly tuned SolarFlare solution would be higher performing -- especially on the same machine with TCP-loopback acceleration. But you don't know until you try it...

edit: added note about epoll

Re: How Fast can A Single Instance of Redis be?

#59

Post by hinkley »

specialist wrote: post #25

I'd like to see the client code, see how it manages backpressure. (I'm probably being thick; the benchmark code is probably linked and I'm just not seeing it.) I recently maintained some nodejs & expressjs stuff. Neither the Redis clients (or the original developers of our stuff) have any concept of backpressure (throttling). In our case, HTTP request received would cause a Redis request. For whatever reason, express…

There’s a memcached client that uses connection pools but it has its own set of problems.

And you have to use nginx plus to get the backpressure support on the ingest side. If my levels of capability and capacity were a little higher I’d be tempted to start writing a reverse proxy in Rust.

Re: How Fast can A Single Instance of Redis be?

#60

Post by hinkley »

ariosto wrote: post #44
specialist wrote: post #25

I'd like to see the client code, see how it manages backpressure. (I'm probably being thick; the benchmark code is probably linked and I'm just not seeing it.) I recently maintained some nodejs & expressjs stuff. Neither the Redis clients (or the original developers of our stuff) have any concept of backpressure (throttling). In our case, HTTP request received would cause a Redis request. For whatever reason, express…

We've been running into the issue where our redis instance(s) randomly dies. I haven't been able to pin point the problem (nodejs+redis). Would love to hear your thoughts on some gotchas to look out for.

Not OP, but I wonder what the collectd plugin for Redis would tell you.
Post reply on HN