Live data from Hacker News

Apple open-sources FoundationDB

foundationdb.org

361–370 of 453 posts

Re: Apple open-sources FoundationDB

#361
post #356
post #353

Earlier quoted context omitted.

On the network side no, it's much more crappy on AWS.

Which provider is the best, network wise?

I only have experience with AWS and on prem and high quality colo like Equinix. Possibly due to reduced complexity and having full control over networking setup but significantly fewer issues vs AWS.

Re: Apple open-sources FoundationDB

#362

I am not a database expert by any means but have been curious about distributed data systems and had not heard of FoundationDB till now and was very excited to read about it. On reading through the documentation, I encountered a section on "Known Limitations"[1] which stated that keys could not be larger than 10kb and values cannot be larger than 100kb. This seems to be a major limitation. Am I missing something or i…

Because the data model is ordered, large blobs can and normally should be mapped to a bunch of adjacent keys and read with a range read, not a single huge value. That also allows you to read or write just part of one efficiently.

Re: Apple open-sources FoundationDB

#363

Earlier quoted context omitted.

Does the trigger execute in scope of triggering transaction with the same isolation level?

Versionstamped operations and transaction logging are fully transactional. Watches are asynchronous: they are used to optimize a polling loop that would "work" without them.

Would versionstamped operations fit for the log abstraction modeling question I've asked about here on the forums?

https://forums.foundationdb.org/t/log-abstraction-on-foundat...

Re: Apple open-sources FoundationDB

#364
post #346

Earlier quoted context omitted.

FoundationDB at its core is not a graph database. You could build a graph database on top of it, using FoundationDB as a very strong and feature rich storage engine, however you'd like. It would be much simpler to do than building a new (especially a distributed) graph database from scratch.

So FDB's role is more comparable to that of, say, RocksDB or LevelDB than an application-level database like Postgres? I didn't really pay much attention to Foundation before Apple bought them and am unsure how it fits in the wider database ecosystem.

You can think of FDB as a distributed storage engine. It has the same low level data model as the engines you mention, but has distributed transactions, fault tolerance, automatic data partitioning, operational tooling, etc built in. So if you build an "application level database" or library on top of it, it is automatically a distributed database.

Re: Apple open-sources FoundationDB

#365
post #89

Noticed the Visual Studio project files. Was Windows a primary development platform for foundationdb?

Before FDB was acquired by Apple, a lot of the engineers used Visual Studio on Windows. It's a good IDE for C++. You certainly don't need it to develop though!

Windows was never the most important deployment platform for the product. I'm not sure how performant the server is. But it should work, and I would think the client is fine.

Re: Apple open-sources FoundationDB

#366
post #118
post #109

Earlier quoted context omitted.

Honest question, does MongoDB work?

Until recently, no. But apparently they got serious about making it work and now it actually does. See https://jepsen.io/analyses/mongodb-3-4-0-rc3 for verification. I realized this 2 months ago at https://news.ycombinator.com/item?id=16386129

That's Nostradamus-level crazytown. What next, PHP strongly enforcing a sound static type system with immutable defaults, GADTs and dependent types?

Congrats to the MongoDB team!

Re: Apple open-sources FoundationDB

#367

Earlier quoted context omitted.

Versionstamped operations and transaction logging are fully transactional. Watches are asynchronous: they are used to optimize a polling loop that would "work" without them.

Would versionstamped operations fit for the log abstraction modeling question I've asked about here on the forums? https://forums.foundationdb.org/t/log-abstraction-on-foundat...

Yes.

Re: Apple open-sources FoundationDB

#368
post #214

Earlier quoted context omitted.

FoundationDB stores 2N+1 copies of some "coordination state" and does a consensus algorithm whenever it is updated. But this state doesn't contain a copy of your data; basically think of it as storing a replication configuration. It's very small and rarely changes. In the happy case, replication takes place using the replicas and quorum rules specified by this configuration. For example, you might require writes to s…

Yup, this works. I wonder if this is the first commercial application of this insight? http://hh360.user.srcf.net/blog/2016/08/majority-agreement-i... Can you tell us what consensus algorithm you're using? Raft, or something else? Your own implementation or something off the shelf?

I'm not sure.

The coordination consensus is (our own implementation of) disk paxos, which we liked for its operational properties in our context (the coordinators don't need to know about each other or communicate directly). An early version of fdb had a dependency on Zookeeper for this purpose; you can use anything.

Re: Apple open-sources FoundationDB

#369
post #278
post #168

Earlier quoted context omitted.

It seems that MUMPS doesn't have serializable isolation.

Yeah, when I read about that I thought it sounded neat—make sure the index updates with the data. On the other hand, I think of CAP theorem as an iron triangle. If you are gaining consistency, what’s the trade off?

This _is_ the usual trade off, but what makes FoundationDB so crazy is that it's a CP system that has a performance profile that AP systems would have a hard time matching.

Re: Apple open-sources FoundationDB

#370

Earlier quoted context omitted.

It's closest to TiDB's key-value layer; a building block for more complex systems. More traditional, monolithic databases like CockroachDB (SQL) or FaunaDB (NoSQL) trade off extensibility for the benefits in performance and operations that come from very tight coupling. In my understanding, FoundationDB's transaction management is closest to FaunaDB's; read/write sets are linearized in memory in preprocessing nodes a…

Wait CockroachDB is monolithic? I though it was distributed.

I think the use of monolithic here is that CockroachDB is higher level than FoundationDB. The project was not designed to be your do-everything-DB that you layer different systems on, but has a specific goal and is intended to be used directly.
Post reply on HN