Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FoundationDB 6.0 released, featuring multi-region support

foundationdb.org

Post reply on HN
Print view View original

61–70 of 72 posts

Re: FoundationDB 6.0 released, featuring multi-region support

#61

Post by lykr0n »

monstrado wrote: post #35
lykr0n wrote: post #24

Earlier quoted context omitted.

My blocker is that the documentation is confusing and not very straight forward. The tutorials are not very verbose and the code (at least for python) is not written in a way that is easy for someone to understand. I'd be very interested in usage for time-series, but it looks like to get a working example up I'd need to fully parse the documentation and tutorials in order to do it as there is no "Let's build a Timese…

Have you given https://apple.github.io/foundationdb/time-series.html a read? Since FoundationDB is lexicographically sorted it's pretty straight forward to keep things sorted for reading data in chronological order. For example, if you want to read "last 5 minutes" of data, you can keep your timestamp at the end of the key in reverse order (Long.MaxValue - timestamp).

Yeah, but that doc assumes you've read and understood everything else in the docs. That document provides no help in understanding the database, it's concepts, or how to implement them. I could try and reconcile that doc with the python tutorial and everything else, but I just want something where I can copy and paste and it works.

FoundationDB doesn't have a batteries included documentation like Clickhouse does: https://clickhouse.yandex/tutorial.html

Re: FoundationDB 6.0 released, featuring multi-region support

#62

Post by rbranson »

continuations wrote: post #56
ryanworl wrote: post #53

Earlier quoted context omitted.

Those first two are pretty trivial. Keys are already stored in sorted order in a b-tree. Secondary indexing involves putting the indexed value to the left of the primary key in the key you write.

> Keys are already stored in sorted order in a b-tree I thought FoundationDB stored data in a hash table. Didn't realize it uses a b-tree. Thanks for the clarification. > Secondary indexing involves putting the indexed value to the left of the primary key in the key you write By secondary index I meant having a sort order that's different from the primary key sort order.

FoundationDB provides the building blocks. Secondary indexes can be built on top of the regular primary index. Transactions make it possible to maintain a consistent 2I using a K/V interface.

Re: FoundationDB 6.0 released, featuring multi-region support

#63

Post by discoball »

ryanworl wrote: post #55
discoball wrote: post #54

How does FDB compare to Spanner as far as the Consistency model and trade offs?

FoundationDB and Spanner both offer external consistency. Spanner does this through synchronized clocks. FoundationDB has a similar clock called TimeKeeper, which is not a clock per se but a counter which advances approximately 1M times per second. Transactions are ordered based on this timestamp.

With a Lamport Clock (counter, logical clock) you could end up with the following due to dependence on conflict resolution aka optimistic MVCC rather than Wall Time (copies do from another HN):

“it is possible for a transaction C to be aborted because it conflicts with another transaction B, but transaction B is also aborted because it conflicts with A (on another resolver), so C "could have" been committed. When Alec Grieser was an intern at FoundationDB he did some simulations showing that in horrible worst cases this inaccuracy could significantly hurt performance. But in practice I don't think there have been a lot of complaints about it.”

Re: FoundationDB 6.0 released, featuring multi-region support

#64

Post by zenbai »

ryanworl wrote: post #4

To anyone who is on the fence about putting FoundationDB into production (or at least evaluating it for their use cases), what is the number one thing you think is missing or you're worried about? i.e. - a SQL interface, - pre-packaged data structure libraries, - monitoring, - limitations of FoundationDB itself, - etc. I'm working on a talk for the upcoming FoundationDB Summit and I'd love to address some real-world…

I am considering FoundationDB for a SaaS project. Here are some questions I have:

1. What is the recommended way to filter on multiple indexes?

2. What is the recommended way to filter on one or more indexes and sort on another or more index(es)

3. If I use FoundationDB as my main datastore, how should I implement full text search?

4. From what I understand, FoundationDB stores keys sorted globally. How does it handle hot spots where a range of keys that is very frequently accessed is on one machine that gets overloaded?

Edit 5. How do you do backups?

Re: FoundationDB 6.0 released, featuring multi-region support

#65

Post by Nullabillity »

ryanworl wrote: post #4

To anyone who is on the fence about putting FoundationDB into production (or at least evaluating it for their use cases), what is the number one thing you think is missing or you're worried about? i.e. - a SQL interface, - pre-packaged data structure libraries, - monitoring, - limitations of FoundationDB itself, - etc. I'm working on a talk for the upcoming FoundationDB Summit and I'd love to address some real-world…

Trust. Apple already killed it once, just after they bought it. How am I supposed to trust that they wouldn't do so again?

A SQL interface might lessen the impact of that somewhat, but then I might as well just use Postgres from the start.

Re: FoundationDB 6.0 released, featuring multi-region support

#66

Post by statictype »

Nullabillity wrote: post #65
ryanworl wrote: post #4

To anyone who is on the fence about putting FoundationDB into production (or at least evaluating it for their use cases), what is the number one thing you think is missing or you're worried about? i.e. - a SQL interface, - pre-packaged data structure libraries, - monitoring, - limitations of FoundationDB itself, - etc. I'm working on a talk for the upcoming FoundationDB Summit and I'd love to address some real-world…

Trust. Apple already killed it once, just after they bought it. How am I supposed to trust that they wouldn't do so again? A SQL interface might lessen the impact of that somewhat, but then I might as well just use Postgres from the start.

It's open source now right?

Re: FoundationDB 6.0 released, featuring multi-region support

#67

Post by ryanworl »

discoball wrote: post #63
ryanworl wrote: post #55

Earlier quoted context omitted.

FoundationDB and Spanner both offer external consistency. Spanner does this through synchronized clocks. FoundationDB has a similar clock called TimeKeeper, which is not a clock per se but a counter which advances approximately 1M times per second. Transactions are ordered based on this timestamp.

With a Lamport Clock (counter, logical clock) you could end up with the following due to dependence on conflict resolution aka optimistic MVCC rather than Wall Time (copies do from another HN): “it is possible for a transaction C to be aborted because it conflicts with another transaction B, but transaction B is also aborted because it conflicts with A (on another resolver), so C "could have" been committed. When Ale…

Yes, I think is fairly well known in the optimistic family of concurrency control algorithms you can get into situations where aborts are not necessary.

https://db.cs.cmu.edu/papers/2016/yu-sigmod2016.pdf

Section 3.4 of this paper covers another example of this.

Re: FoundationDB 6.0 released, featuring multi-region support

#68

Post by ddorian43 »

jared2501 wrote: post #50
realreality wrote: post #36

Earlier quoted context omitted.

- transaction size and duration limitations. I can almost understand the limitation on large write transactions, but the same size limitation applies to read transactions. If you’re doing a large range read, you may not know whether your range will reach the 10MB limit, and thus raise an exception. - the storage backends seem less impressive than the marketing leads you to believe. The default memory backend is obvio…

On shrinking a cluster: you'll want to use the fdbcli to "exclude" nodes. Should be pretty straight forward (search the docs for the word "exclude"). On write amplification: a factor of 3x is not actually that unusual. The default RocksDB size amplification is 2x, and I've seen performant LSM trees with about 3x write amplification. On the single threaded bottleneck: this is an inherent issue you have when you put yo…

Scylladb/redis can also do a lot of calls with single thread/core.

Re: FoundationDB 6.0 released, featuring multi-region support

#69

Post by ryanworl »

ddorian43 wrote: post #68
jared2501 wrote: post #50

Earlier quoted context omitted.

On shrinking a cluster: you'll want to use the fdbcli to "exclude" nodes. Should be pretty straight forward (search the docs for the word "exclude"). On write amplification: a factor of 3x is not actually that unusual. The default RocksDB size amplification is 2x, and I've seen performant LSM trees with about 3x write amplification. On the single threaded bottleneck: this is an inherent issue you have when you put yo…

Scylladb/redis can also do a lot of calls with single thread/core.

FoundationDB single-core performance is fine. From my testing on the memory engine (and the docs), you can expect 70k+ reads/second/core for small keys and values. But crucially this means you must have concurrency to drive throughput.

No database can magically make your serial access pattern faster. Amdahl's law and all that.

FoundationDB's latency for your specific workload is up to how good you are at designing your algorithm for concurrency. If you do every step serially, you'll be spending most of your time waiting for the network.

Re: FoundationDB 6.0 released, featuring multi-region support

#70

Post by Nullabillity »

statictype wrote: post #66
Nullabillity wrote: post #65

Earlier quoted context omitted.

Trust. Apple already killed it once, just after they bought it. How am I supposed to trust that they wouldn't do so again? A SQL interface might lessen the impact of that somewhat, but then I might as well just use Postgres from the start.

It's open source now right?

Sure. That doesn't mean they couldn't do a lot of damage by taking down the documentation, official repo, and download mirrors.

Again.

Post reply on HN