Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple open-sources FoundationDB

foundationdb.org

Post reply on HN
Print view View original

291–300 of 453 posts

Re: Apple open-sources FoundationDB

#291

Post by throwaway84742 »

eropple wrote: post #21
throwaway84742 wrote: post #16

Earlier quoted context omitted.

Google Spanner? Not OSS, but globally consistent, scalable, and extensively battle tested.

Spanner can do global consistency and (some?) transactions but I'm unaware of it being able to do the sort of layering Foundation can internally to expose largely different database forms on top of it. I have never used Spanner, though, so I'm open to being corrected.

What do you mean by “forms”? Spanner is also layered. The bottom layer is basically a key-value store. On top of that there’s a full blown SQL layer, which, BTW can work with hierarchical records as well as flat tables. Both support transactions and guarantee global consistency.

Re: Apple open-sources FoundationDB

#292

Post by zbentley »

dboreham wrote: post #111
itp wrote: post #40

Earlier quoted context omitted.

Speaking as the original author of this monstrosity of a build system, please be careful before offering praise here. To be clear, there is a top-level, non-recursive Makefile that uses the second expansion feature of GNU make, translating Visual Studio project files into generated Makefile inputs that are transformed into targets to power the build. Although it starts by running `make`, it's about as in-house as a t…

This kind of deep-inside-baseball from-the-horses-mouth interaction is what's so awesome about HN!

> deep-inside-baseball from-the-horses-mouth

Does the horse choke on the baseball? Is there an equine version of the Heimlich maneuver to be performed on horses suffering from mixed-metaphorical-adage-induced asphyxiation?

Re: Apple open-sources FoundationDB

#293

Post by spullara »

josephg wrote: post #283
voidmain wrote: post #263

Earlier quoted context omitted.

Yes. I don't know how well documented it is, but there is an API (well, system keyspace) that can configure the database to log transactions for a selected key range (up to and including the whole database) into another selected key range. It is used by backup and asynchronous replication tools. The format of the configuration keys and transaction logs should be considered less stable than the core key/value API, whi…

> Alternatively, your application or layer can use the "versionstamp" atomic operations to write its own ordered log of what it is doing, or other indexing tricks. I'm not sure I understand. Are you suggesting having a second key space at `ops/{VERSIONSTAMP}` or something where values contain enough information about the operation to be able to process changes in an indexer? The indexer could then clean up after itse…

Yes.

Re: Apple open-sources FoundationDB

#294

Post by tmoertel »

voidmain wrote: post #46
openasocket wrote: post #10

I hadn't heard of FoundationDB before, so I did some digging into the features: https://apple.github.io/foundationdb/features.html . It seems to claim ACID transactions with serializable isolation, but also says later on that it uses MVCC, slower clients won't slow down operations, and that it allows true interactive queries. I didn't think an MVCC implementation could provide that level of isolation, and I'm not eve…

I'll try to give you a quick introduction. The architecture talk I recorded for new engineers working on the product ran to four or five hours, I think :-). In short, it is serializable optimistic MVCC concurrency. A FDB transaction roughly works like this, from the client's perspective: 1. Ask the distributed database for an appropriate (externally consistent) read version for the transaction 2. Do reads from a cons…

Does the implementation handle the case that you want to do a write that is conditioned on a prior read finding no corresponding record(s)?

Re: Apple open-sources FoundationDB

#295

Post by voidmain »

josephg wrote: post #283
voidmain wrote: post #263

Earlier quoted context omitted.

Yes. I don't know how well documented it is, but there is an API (well, system keyspace) that can configure the database to log transactions for a selected key range (up to and including the whole database) into another selected key range. It is used by backup and asynchronous replication tools. The format of the configuration keys and transaction logs should be considered less stable than the core key/value API, whi…

> Alternatively, your application or layer can use the "versionstamp" atomic operations to write its own ordered log of what it is doing, or other indexing tricks. I'm not sure I understand. Are you suggesting having a second key space at `ops/{VERSIONSTAMP}` or something where values contain enough information about the operation to be able to process changes in an indexer? The indexer could then clean up after itse…

Yes. Or if it's not important for the indexer to process things chronologically, you could just have an index of the primary key (only) of records that haven't been indexed.

If you are trying to make your external index MVCC, then you will want to carry some version information too.

This kind of question might be better served by the new community forum you can get to from the website!

Re: Apple open-sources FoundationDB

#297

Post by gnat »

mattkrea wrote: post #286
mercutio2 wrote: post #270

Earlier quoted context omitted.

Well, clangd is a Google project, which Apple has decided to start contributing to, so probably doesn’t belong on your list. Apple, like everyone else, wants to commoditize their complements.

Clang was an Apple project from the start.. I'm not sure what is telling you it is a Google project

In case a citation is needed: https://en.wikipedia.org/wiki/Clang

> Apple chose to develop a new compiler front end from scratch, supporting C, Objective-C and C++. This "clang" project was open-sourced in July 2007.

Re: Apple open-sources FoundationDB

#298

Post by deepanchor »

thx4thefish wrote: post #287
thx4thefish wrote: post #267

This seems nice. Besides a bunch of fanboy comments coming from the creators and devs, why is this exciting to the rest of us where things like the capability to join tables in an rdbms is trivial.

Thanks for the downvote. My question still remains.

Didn't downvote you, but I believe the excitement is over FoundationDB's ability to perform ACID compliant distributed transactions without sacrificing performance -which to my knowledge no current RDBMS or even NoSql can do.

Re: Apple open-sources FoundationDB

#299

Post by notacoward »

jamesblonde wrote: post #121
wwilson wrote: post #2

This is INCREDIBLE news! FoundationDB is the greatest piece of software I’ve ever worked on or used, and an amazing primitive for anybody who’s building distributed systems. The short version is that FDB is a massively scalable and fast transactional distributed database with some of the best testing and fault-tolerance on earth[1]. It’s in widespread production use at Apple and several other major companies. But the…

I am one of the designers of probably the best known metadata storage engine for a distributed filesystem, hopsfs - www.hops.io. When I looked at FoundationDB before Apple bought you, you supported transactions - great. But we need much more to scale. Can you tell me which of the following you have: row-level locks partition-pruned index scans non-serialized cross-partition transactions (that is, a transaction coordi…

> best known metadata storage engine for a distributed filesystem, hopsfs

Not even close. I don't even see anything I'd call a filesystem mentioned on your web page. I missed FAST this year, and apparently you had a paper about using Hops as a building block for a non-POSIX filesystem - i.e. not a filesystem in my and many others' opinion - but it's not clear whether it has ever even been used in production anywhere let alone become "best known" in that or any other domain. I know you're proud, perhaps you have reason to be, but please.

Re: Apple open-sources FoundationDB

#300

Post by elcritch »

jamesblonde wrote: post #179
voidmain wrote: post #151

Earlier quoted context omitted.

It's somewhat hard to answer your questions because the architecture (and hence, terminology) of FoundationDB is a little different than I think you are used to. But I will give it a shot. FoundationDB uses optimistic concurrency, so "conflict ranges" rather than "locks". Each range is a (lexicographic) interval of one or more keys read or written by a transaction. The minimum granularity is a single key. FoundationD…

Thanks for the detailed answer. Is it actually serializable isolation - does it handle write skew anomalies ( https://en.wikipedia.org/wiki/Snapshot_isolation )? Most OCC systems I know have only snapshot isolation. Systems that sound closest to FoundationDB's transaction model that i can think of are Omid ( https://omid.incubator.apache.org/ ) and Phoenix ( https://phoenix.apache.org/transactions.html ). They both s…

I agree with voidmain’s comment as secondary indexes shouldn’t be any different than the primary KV in your case. Almost seems that you’re focusing on a SQL/Relational database architecture but storing your data demoralized anyways. Odd combination of thoughts.
Post reply on HN