Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

KeyDB CEO Interview: Getting into YC with a Fork of Redis

console.dev

Post reply on HN
Print view View original

71–77 of 77 posts

Re: KeyDB CEO Interview: Getting into YC with a Fork of Redis

#71

Post by injinj »

atat7024 wrote: post #70
injinj wrote: post #56

Earlier quoted context omitted.

No, I don't believe that's what it means in this case. I took the usage to mean that since Rust has memory model which is different enough from C to require a redesign. I looked at many implementations of Redis and read many KV papers. My redesign reason was similar to a "clean room Rust" reason, I desired a memory model that used shared memory that was independent of the protocol (Redis RESP in this case), allowing…

"Rewrite", then.

Yeah. "Refactor" may be more accurate.

Re: KeyDB CEO Interview: Getting into YC with a Fork of Redis

#72

Post by PeterCorless »

ddorian43 wrote: post #32
nemothekid wrote: post #18

Earlier quoted context omitted.

It's been a long time since I've looked at KeyDB, but IIRC KeyDB is just Redis plus a spinlock. It's actually still very performant. There are other "toy" reimplementations of Redis in Rust that take the same approach and aren't even as performant as single threaded Redis. The next approach you could take is using something like Glommio and take a thread-per-core design to Redis. I think that approach has a lot of po…

RonDB (NDB Cluster) takes a different approach to threading and claims it's faster than scylladb-style sharding http://mikaelronstrom.blogspot.com/2021/03/designing-thread-...

"ScyllaDB also handles complex query processing in the same engine. RonDB does all complex query processing in MySQL in a separate program." — i.e., as soon as things might get a bit hairy, RonDB punts it off-system. While a clever way to keep performance numbers high, you'd have to also then analyze the MySQL latencies and throughputs to get an overall view of complete system performance.

It's a clever dodge, but it's not a magical way to eliminate the need for those pesky CPU cycle times.

Re: KeyDB CEO Interview: Getting into YC with a Fork of Redis

#73

Post by ddorian43 »

PeterCorless wrote: post #72
ddorian43 wrote: post #32

Earlier quoted context omitted.

RonDB (NDB Cluster) takes a different approach to threading and claims it's faster than scylladb-style sharding http://mikaelronstrom.blogspot.com/2021/03/designing-thread-...

"ScyllaDB also handles complex query processing in the same engine. RonDB does all complex query processing in MySQL in a separate program." — i.e., as soon as things might get a bit hairy, RonDB punts it off-system. While a clever way to keep performance numbers high, you'd have to also then analyze the MySQL latencies and throughputs to get an overall view of complete system performance. It's a clever dodge, but it…

Look a little closer Peter, I doubt you arrived that fast at that conclusion being right.

It's not a dodge, the MySQL process is in the same server and can even use shared memory.

He claims in total to be better.

Re: KeyDB CEO Interview: Getting into YC with a Fork of Redis

#75

Post by avinassh »

jdsully wrote: post #67
avinassh wrote: post #65

Earlier quoted context omitted.

can you share more details about it? I thought locks are slow, but despite locks the is latency halved here.

KeyDB uses a custom lock I wrote called the “fastlock”. At its core it’s a ticket lock with some tweaks to better tune it to KeyDB. When uncontested it’s a single atomic increment. When contested ownership changes extremely fast. If we spin too long the thread will sleep although we wait much longer than any other lock you’ll find. When I first tried making KeyDB I used posix locks. Those were way too slow.

I am too newb to understand half of those terms :)

Do you have any recommendations for a reading material to understand the underlying concepts?

Re: KeyDB CEO Interview: Getting into YC with a Fork of Redis

#76

Post by jdsully »

avinassh wrote: post #75
jdsully wrote: post #67

Earlier quoted context omitted.

KeyDB uses a custom lock I wrote called the “fastlock”. At its core it’s a ticket lock with some tweaks to better tune it to KeyDB. When uncontested it’s a single atomic increment. When contested ownership changes extremely fast. If we spin too long the thread will sleep although we wait much longer than any other lock you’ll find. When I first tried making KeyDB I used posix locks. Those were way too slow.

I am too newb to understand half of those terms :) Do you have any recommendations for a reading material to understand the underlying concepts?

The wikipedia page on ticket locks has a good overview: https://en.wikipedia.org/wiki/Ticket_lock

Generally speaking you don't want to be dealing with this stuff unless you really have to. For 99.999% of people the locking primitives that come with your programming language are good enough.

Re: KeyDB CEO Interview: Getting into YC with a Fork of Redis

#77

Post by avinassh »

jdsully wrote: post #76
avinassh wrote: post #75

Earlier quoted context omitted.

I am too newb to understand half of those terms :) Do you have any recommendations for a reading material to understand the underlying concepts?

The wikipedia page on ticket locks has a good overview: https://en.wikipedia.org/wiki/Ticket_lock Generally speaking you don't want to be dealing with this stuff unless you really have to. For 99.999% of people the locking primitives that come with your programming language are good enough.

I went through the wiki, certainly it was interesting. I believe the implementation is at fastlock.cpp [0], I will go through it. You said earlier:

> If we spin too long the thread will sleep although we wait much longer than any other lock you’ll find.

Did you do any tweaks to mitigate this?

> Generally speaking you don't want to be dealing with this stuff unless you really have to.

Ofcourse. I am just way too curious and excited to learn about these!

[0] - https://github.com/EQ-Alpha/KeyDB/blob/v6.0.16/src/fastlock....

Post reply on HN