Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pebble: A RocksDB Inspired Key-Value Store Written in Go

cockroachlabs.com

Post reply on HN
Print view View original

71–80 of 84 posts

Re: Pebble: A RocksDB Inspired Key-Value Store Written in Go

#71

Post by paulez »

willvarfar wrote: post #24

I’ve run into serious house burning down problems with myrocks too. Simple recipe to crash MySQL in a way that is unrecoverable: do ALTER TABLE on a big table and it runs out of RAM, crashes, and refuses to restart, ever. Googling and people have been reporting the error on restarting several times on lists and things. What help is it to report to Maria dB or something? But do FB notice? Seems not. Here’s hoping some…

That's a long standing problem with MySQL, DDL statements (ALTER TABLE and such) where not atomic until version 8.0. This required some serious work on InnoDB as you can read some details at https://mysqlserverteam.com/atomic-ddl-in-mysql-8-0/

Re: Pebble: A RocksDB Inspired Key-Value Store Written in Go

#73

Post by petermattis »

dis-sys wrote: post #61
petermattis wrote: post #45

Earlier quoted context omitted.

Yes. So far performance was worse in experiments, and the durability improvements are questionable because it is extremely difficult to get a clear understanding of the durability semantics of direct IO. If you can find a pointer to clear documentation of what those semantics are I'd be extremely interested in reading it.

RocksDB supports using Direct IO for flush and compaction (use_direct_io_for_flush_and_compaction), enabling that can improve write throughput for my workload in RocksDB. Any plan to do that in pebble?

Direct IO is on our radar, though when I experimented with enabling direct IO in RocksDB it only hurt CockroachDB benchmarks. This was several years ago. I believe newer releases of RocksDB have made improvements in this area.

Re: Pebble: A RocksDB Inspired Key-Value Store Written in Go

#74

Post by m-p-3 »

_ea1k wrote: post #25
chaosharmonic wrote: post #2

Was anyone else deeply saddened about three words into the headline, on realizing this wasn't a watch? (RIP)

I definitely was. I still use one now, more than 3 years after their business failure. I really wish someone would make something like the Pebble Time 2.

I'd still use my Time Steel if the battery didn't die, and I'm afraid of breaking it and its waterproofing if I attempt a replacement.

Re: Pebble: A RocksDB Inspired Key-Value Store Written in Go

#75

Post by strken »

ecnahc515 wrote: post #26
dfee wrote: post #5

As a consumer, why would I want something like this written in Go vs. Rust? Is it just that Rust is really good with developer relations? Because it feels like to me that all new foundational technology is safer and faster in a language like Rust, and things written in Go should be higher up the food chain.

In addition to what others mentioned, even if Cockroach is written in Go, they could have used Rust, but the trade-off is that they would need to use cgo which introduces extra complexity for building, debugging, and has performance trade-offs that a pure-Go based solution doesn't necessarily have.

Reading this, I wonder how hard it would be to write or generate assembly bindings for libraries like rocksdb and sqlite, and whether you'd be able to do any better than cgo.

Re: Pebble: A RocksDB Inspired Key-Value Store Written in Go

#76

Post by strken »

brokencode wrote: post #49
dfee wrote: post #33

Earlier quoted context omitted.

I really don’t understand the downvotes. I’m not experienced with either language - and this has nothing to do with a flame war. The question, unstated and unopinionated AND intellectually honest was: does language impact community adoption - and if so, what are the drivers behind it. If I were going to write a foundational technology, I probably wouldn’t write it in NodeJS, not that it couldn’t be done, but because…

I don’t think Cockroach cares about adoption. This is not meant to be a generally useful product in itself outside of their database. So the language was chosen mainly based on their familiarity with Go and its ability to integrate with their existing codebase. I think their omission of major features such as transactions is more likely to limit adoption than the language choice, so language choice is kind of irrelev…

The integration with their existing codebase might benefit from an explanation. In Go, calls to C libraries are done using a compiler feature called cgo, and there's a performance penalty that the cockroachdb team measured at 171ns per call, plus you sometimes have to copy additional data around for memory safety[0]. So far as I know, there's no way to avoid this penalty that doesn't involve building a tool that does the same thing as cgo.

In this case, given that they already have a database written in Go, writing the backing kv store library in Go has a clear performance benefit of 171ns every operation. This is above and beyond just familiarity and easy integration, although those are also important.

[0] https://www.cockroachlabs.com/blog/the-cost-and-complexity-o...

Re: Pebble: A RocksDB Inspired Key-Value Store Written in Go

#77

Post by petermattis »

strken wrote: post #75
ecnahc515 wrote: post #26

Earlier quoted context omitted.

In addition to what others mentioned, even if Cockroach is written in Go, they could have used Rust, but the trade-off is that they would need to use cgo which introduces extra complexity for building, debugging, and has performance trade-offs that a pure-Go based solution doesn't necessarily have.

Reading this, I wonder how hard it would be to write or generate assembly bindings for libraries like rocksdb and sqlite, and whether you'd be able to do any better than cgo.

Cgo exists because goroutine stacks are small and grow on demand. C/C++ expect large fixed size stacks. While you can technically call into C++ without going through cgo (see https://github.com/petermattis/fastcgo), you can't avoid the stack problem. Cgo solves it by switch from the goroutine stack to the stack of the thread underlying the goroutine.

Re: Pebble: A RocksDB Inspired Key-Value Store Written in Go

#78

Post by yoshinorim »

willvarfar wrote: post #24

I’ve run into serious house burning down problems with myrocks too. Simple recipe to crash MySQL in a way that is unrecoverable: do ALTER TABLE on a big table and it runs out of RAM, crashes, and refuses to restart, ever. Googling and people have been reporting the error on restarting several times on lists and things. What help is it to report to Maria dB or something? But do FB notice? Seems not. Here’s hoping some…

I am one of the creators of MyRocks at FB. We have a few common MySQL features/operations we don't use at FB. Notably:

1) Schema Changes by DDL (e.g. ALTER TABLE, CREATE INDEX)

2) Recovering primary instances without failover

We use our own open source tool OnlineSchemaChange to do schema changes (details: https://github.com/facebook/mysql-5.6/wiki/Schema-Changes), which is heavily optimized for MyRocks use cases like utilizing bulk loading for both primary and secondary keys. ALTER TABLE / CREATE INDEX support in MyRocks is limited and suboptimal -- it does not support Online/Instant DDL (so blocking writes to the same table during ALTER), and enters non bulk loading path and trying to load the entire table in one transaction -- which may hit row lock count limit or out of memory. We have plans to improve regular DDL paths in MyRocks in MySQL 8.0, including supporting atomic, online and instant schema changes.

I am also realizing that a lot of external MySQL users still don't have auto failover and try to recover primary instances if they go down. This means single instance availability and recoverability is much more important for them. We set rocksdb_wal_recovery_mode=1 (kAbsoluteConsistency) by default in MyRocks, which actually degraded recoverability (higher chances to refuse to start even if it can be recovered from binlog). We're changing defaults to 2 (kPointInTimeRecovery) so that it can be more robust without relying on replicas for recovery.

It would have been a really bad experience when hitting OOM by 1) then failing to restart because of 2). We have relations with MariaDB and Percona, and will make default behavior better for users.

Re: Pebble: A RocksDB Inspired Key-Value Store Written in Go

#79

Post by diehunde »

itsbilal wrote: post #15
malisper wrote: post #3

So I understand the rationale for writing your own storage layer and think this is an awesome project, but there's something missing for me. One of the issues Peter brings up is they've come across a number of serious bugs in RocksDB. My question is, why would Pebble have less bugs. In fact, I would expect it to have significantly more bugs because Coackroach is the only company using Pebble. They mention briefly how…

Hi, I'm on the team that works on Pebble. Partially synced WAL records are easier to detect, as they would just appear as corrupt records and we can stop WAL replay at that point. Non-WAL writes are even easier to handle as SSTable files are immutable once fully written and synced. We rely pretty heavily on fsync/fdatasync calls to guarantee that "all" the data in a given range made it. In addition to randomized cras…

Hi! Awesome work.

One of the cool things about the LevelDB codebase is the `Repository contents` section. A brief description of the most relevant modules so people can get familiarized with the code base quicker. As someone very interested in storage engines, I would love to see something similar here. Are you guys planning on adding some extra documentation to the project?

Re: Pebble: A RocksDB Inspired Key-Value Store Written in Go

#80

Post by scott_s »

pdpi wrote: post #20
d--b wrote: post #4

Earlier quoted context omitted.

Well, I think what they're saying is that they'd rather have bugs in code they've written than in code that is written by other people and in another language, and for which they don't control the patching pipeline. If RocksDB had had no bugs, they wouldn't have needed to write Pebble.

That's an argument for them using it, but it's also basically arguing why nobody else should.

Yes, but I think CockroadDB is in also in a position most other people are not: they are a database, so data storage is not only their expertise, it's their reason for existing. They have the people, the expertise, and it's core to their business. Most people don't have that, so they can't justify writing their own storage engine.
Post reply on HN