Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pebble: A RocksDB Inspired Key-Value Store Written in Go

cockroachlabs.com

Post reply on HN
Print view View original

51–60 of 84 posts

Re: Pebble: A RocksDB Inspired Key-Value Store Written in Go

#51

Post by jasonzemos »

petermattis wrote: post #41
jasonzemos wrote: post #32

Concurrency and multithreading are a major focus of both Go and RocksDB. This introduction makes little mention of those areas, and I'm curious if there's any more to be said on this. The article lists several features being reimplemented, including: > Basic operations: Set, Get, Merge, Delete, Single Delete, Range Delete It makes no mention of RocksDB's MultiGet/MultiRead -- is CockroachDB/Pebble limited to query-at…

Pebble does not currently implement MultiGet as CockroachDB did not use RocksDB's MultiGet operation. CockroachDB can use multiple nodes to process a query by decomposing SQL queries along data boundaries and shipping the query parts to be executed next to the data. CockroachDB can't directly use MultiGet because that API was not compatible with how CockroachDB reads keys. RocksDB MultiGet is interesting. Parallelism…

Indeed the conceptual fork point mentioned is RocksDB 6.2.1 which came before those features. The problem with RocksDB is that one thread only makes one request at a time. I should've phrased my question more succinctly: Is Pebble/CockroachDB capable of saturating the backplane with requests in parallel? Does it multiplex a single query by dispatching smaller requests to a thread-pool?

Re: Pebble: A RocksDB Inspired Key-Value Store Written in Go

#52

Post by petermattis »

jasonzemos wrote: post #51
petermattis wrote: post #41

Earlier quoted context omitted.

Pebble does not currently implement MultiGet as CockroachDB did not use RocksDB's MultiGet operation. CockroachDB can use multiple nodes to process a query by decomposing SQL queries along data boundaries and shipping the query parts to be executed next to the data. CockroachDB can't directly use MultiGet because that API was not compatible with how CockroachDB reads keys. RocksDB MultiGet is interesting. Parallelism…

Indeed the conceptual fork point mentioned is RocksDB 6.2.1 which came before those features. The problem with RocksDB is that one thread only makes one request at a time. I should've phrased my question more succinctly: Is Pebble/CockroachDB capable of saturating the backplane with requests in parallel? Does it multiplex a single query by dispatching smaller requests to a thread-pool?

> Is Pebble/CockroachDB capable of saturating the backplane with requests in parallel?

Yes.

> Does it multiplex a single query by dispatching smaller requests to a thread-pool?

Yes, though it depends on the query. Trivial queries (i.e. single-row lookups) are executed serially as that is the fastest way to execute them. Complex queries are decomposed along data boundaries and the query parts are executed in parallel next to where the data is located.

Re: Pebble: A RocksDB Inspired Key-Value Store Written in Go

#53

Post by nitrobeast »

willvarfar wrote: post #24

I’ve run into serious house burning down problems with myrocks too. Simple recipe to crash MySQL in a way that is unrecoverable: do ALTER TABLE on a big table and it runs out of RAM, crashes, and refuses to restart, ever. Googling and people have been reporting the error on restarting several times on lists and things. What help is it to report to Maria dB or something? But do FB notice? Seems not. Here’s hoping some…

Likely your db configuration is very different from what FB uses in production, so they have no incentive to investigate or fix.

Re: Pebble: A RocksDB Inspired Key-Value Store Written in Go

#54

Post by shakna »

AtlasBarfed wrote: post #27

Why would someone remove a non-GC database engine with a database engine with GC? Has Go evolved better low-GC features? As I understand Go GC vs JVM GC, Go avoids major GC by simply pushing it to the future and consuming memory more readily. But a database is a long-running program, so you have to pay the piper eventually.

Upthread [0], you can find some notes and an explanation from one of the devs that they actively work around the GC, because the approach it uses just... Doesn't work for this kind of workload.

> This percentage can be configured by the GOGC environment variable or by calling debug.SetGCPercent. The default value is 100, which means that GC is triggered when the freshly allocated data is equal to the amount of live data at the end of the last collection period. This generally works well in practice, but the Pebble Block Cache is often configured to be 10s of gigabytes in size. Waiting for 10s of gigabytes of data to be allocated before triggering a GC results in very large Go heap sizes.

[0] https://news.ycombinator.com/item?id=24485931

Re: Pebble: A RocksDB Inspired Key-Value Store Written in Go

#55

Post by dilyevsky »

ohnoesjmr wrote: post #48
cube2222 wrote: post #6

How does this compare to Badger[0], another similar in nature key-value store in Go? What were the trade-offs which made it necessary to create something new instead of adapting what exists? [0]: https://github.com/dgraph-io/badger

Badger is written by mad people from my point of view, who disabled issues on github, from my understanding declared it as "done" and "bug free", and any issue tracking is now done on the forum where the threads roll off to the void with no further trace.

Wait really? That’s both hilarious and disappointing

Re: Pebble: A RocksDB Inspired Key-Value Store Written in Go

#56

Post by mrjn »

ohnoesjmr wrote: post #48
cube2222 wrote: post #6

How does this compare to Badger[0], another similar in nature key-value store in Go? What were the trade-offs which made it necessary to create something new instead of adapting what exists? [0]: https://github.com/dgraph-io/badger

Badger is written by mad people from my point of view, who disabled issues on github, from my understanding declared it as "done" and "bug free", and any issue tracking is now done on the forum where the threads roll off to the void with no further trace.

Wao. You describe us as “mad people” because we choose to not use GitHub issues? Is that all it takes to dismiss an open-source software and badmouth its authors? You have gone really low on this.

All the issues have been ported over to Discourse. And no one has declared Badger, bug-free. I don’t know where you got that idea.

Re: Pebble: A RocksDB Inspired Key-Value Store Written in Go

#58

Post by dis-sys »

cube2222 wrote: post #6

How does this compare to Badger[0], another similar in nature key-value store in Go? What were the trade-offs which made it necessary to create something new instead of adapting what exists? [0]: https://github.com/dgraph-io/badger

I looked into the source code of both badger and pebble during the lockdown. From what I learned, they don't operate on the same level. I don't want to publicly bad mouth any open source software, but if you just spend 15 minutes on the source code of pebble and badger, you wouldn't be asking the same question.

Re: Pebble: A RocksDB Inspired Key-Value Store Written in Go

#59

Post by Polylactic_acid »

m-p-3 wrote: post #28
_ea1k wrote: post #25

Earlier quoted context omitted.

I definitely was. I still use one now, more than 3 years after their business failure. I really wish someone would make something like the Pebble Time 2.

I have an Amazfit Bip, but the UI isn't as good as Pebble sadly. There is some work in making a similar OS called RebbleOS[1] currently ongoing. [1]: https://github.com/pebble-dev/RebbleOS Hopefully it will be portable to other low-end smartwatches.

3 years later and it looks like they are just at the trying to get hardware features accessible. At this rate all pebble devices will have died / been discarded before its able to show a notification on your wrist.

Re: Pebble: A RocksDB Inspired Key-Value Store Written in Go

#60

Post by volument »

dis-sys wrote: post #58
cube2222 wrote: post #6

How does this compare to Badger[0], another similar in nature key-value store in Go? What were the trade-offs which made it necessary to create something new instead of adapting what exists? [0]: https://github.com/dgraph-io/badger

I looked into the source code of both badger and pebble during the lockdown. From what I learned, they don't operate on the same level. I don't want to publicly bad mouth any open source software, but if you just spend 15 minutes on the source code of pebble and badger, you wouldn't be asking the same question.

Really curious to hear your findings! We're happy users of Badger, but we have never looked it's internals. I guess you can list the differences you've found without using a bad mouth. Thanks in advance.
Post reply on HN