Apple open-sources FoundationDB • Hacker News Board

Re: Apple open-sources FoundationDB

#441

Post by jkaplowitz » Tue, Apr 24, 2018, 5:00 PM UTC

skissane wrote: post #393

diaz wrote: post #351

Earlier quoted context omitted.

Unless people want to change the mongodb code that they would be using, using the agpl software should be a non issue and there are not problems with it. People should start understanding the available licenses instead of spreading fear.

It isn't "spreading fear", it is just reality. Google bans AGPL-licensed software internally: https://opensource.google.com/docs/using/agpl-policy/ I know that multiple other companies have a similar policy (either a complete ban on using AGPL-licensed software, or special approval required to use it), although unlike Google, they don't post their internal policy publicly. If someone works at one of these companies,…

The main reason it's a problem at many of the companies which ban it is they have a lot of engineers who readily patch and combine code from disparate sources and might not always apply the right discipline to keep AGPL things appropriately separate. Bright-line rules can be quite useful.

It is true that MongoDB's AGPL contagion and compliance burden, if you don't modify it, is less than many fear. It is also true that those corporate concerns are valid. MongoDB does sell commercial licenses so that such companies can handle their unavoidable MongoDB needs, but they would tend to minimize that usage.

Re: Apple open-sources FoundationDB

#442

Post by evanweaver » Tue, Apr 24, 2018, 5:56 PM UTC

cooervo wrote: post #345

evanweaver wrote: post #164

Earlier quoted context omitted.

It's closest to TiDB's key-value layer; a building block for more complex systems. More traditional, monolithic databases like CockroachDB (SQL) or FaunaDB (NoSQL) trade off extensibility for the benefits in performance and operations that come from very tight coupling. In my understanding, FoundationDB's transaction management is closest to FaunaDB's; read/write sets are linearized in memory in preprocessing nodes a…

Wait CockroachDB is monolithic? I though it was distributed.

I mean it in the terms of monolithic process vs. service-oriented architecture, distinct from a distributed vs. centralized operational topology.

FaunaDB and CockroachDB are implemented as monolithic processes and can break encapsulation boundaries for performance reasons. For example, FaunaDB does aggressive predicate pushdown to accelerate intersections and joins, which you cannot do if you have to conform to a key/value interface exclusively. It can also eliminate all network overhead for query data that's local to the processing node.

I understand how that terminology is confusing though...how would you explain it?

Re: Apple open-sources FoundationDB

#443

Post by mvc » Wed, Apr 25, 2018, 8:17 AM UTC

wll wrote: post #332

derefr wrote: post #248

Earlier quoted context omitted.

> you should be able to take Lucene's nice fast immutable data structure and stuff blocks of it (at the term level or below) into FDB values very efficiently. That sounds a lot like Datomic's "Storage Resource" approach, too! Would Datomic-on-FDB make sense, or is there a duplication of effort there?

It most definitely would. Datomic’s single-writer system requires conditional put (CAS) for index and (transaction) log (trees) roots pointers (mutable writes), and eventual consistency for all other writes (immutable writes) [0]. I would go as far as saying a FoundationDB-specific Datomic may be able to drop its single-writer system due to FoundationDB’s external consistency and causality guarantees [1], drop its 64…

Can't see datomic itself ever doing that though because they'd have to support those features in all the backends.

Re: Apple open-sources FoundationDB

#444

Post by mogui » Thu, Apr 26, 2018, 6:57 PM UTC

voidmain wrote: post #440

mogui wrote: post #435

Earlier quoted context omitted.

In case (c) or (d) how can a layer leverage the distributed facilities that FDB gives? I mean if I have clients that connect to a "layer service" that is the one who talks to FDB, I have to manage "layer service" scalabily, fault tolerance etc... by myself.

Yes, and that's the main advantage of choosing (a) or (b). But it's not quite as hard as it sounds; since all your state is safely in fdb you "just" have to worry about load balancing a stateless service.

got it, what will you suggest to do something like that? a simple RPC with a good async framework I've read, like what? an RPC service on top of Twisted for python, similar things in other languages?

thanks :)

Re: Apple open-sources FoundationDB

#445

Post by qaq » Fri, Apr 27, 2018, 3:06 AM UTC

Joeri wrote: post #413

qaq wrote: post #201

Earlier quoted context omitted.

"but we have lost machines, connectivity, seen kernel panics, EBS failures, SSD failures, etc., your usual day in AWS " <=== This I wish more people realized that is a day to day reality if you are in AWS at scale.

The best way I've heard it described is "complex systems run in degraded mode". https://cdn.chrisshort.net/How-Complex-Systems-Fail.pdf Basically once a system is complex enough some part if it is always broken. The software must be designed from the assumption that the system is never running flawlessly.

No doubt but that's pretty high overhead for many projects colo is actually a decent choice but I guess that's not a popular opinion.

Re: Apple open-sources FoundationDB

#446

Post by benmmurphy » Fri, Apr 27, 2018, 4:14 PM UTC

polskibus wrote: post #415

voidmain wrote: post #410

Earlier quoted context omitted.

I explain the basics of our concurrency control here: https://news.ycombinator.com/item?id=16877950 I guess textbook SSI is willing to "reorder" conflicting transactions if the result is still serializable, which could violate external consistency if you don't have any other bounds on the order. In the language of SSI, fdb simply aborts the later of any pair of read/write transactions with an rw-conflict, in accordan…

as far as I know SSI as implemented in Postgresql, aborts the way you describe, as per https://drkp.net/papers/ssi-vldb12.pdf

postgresql is not strictly serializable/externally consistent

for example this will commit under serializable:

    create table counters(counter int);

    insert into counters(counter) values(1);

    BEGIN TRANSACTION ISOLATION LEVEL serializable; 
    select sum(counter) from counters; 
    /* insert sum into counters. wait until committing next transaction before executing the insert */
    insert into counters(counter) values(1);
    COMMIT;

    /* this transaction should commit before doing the insert in the above transaction and after the above transaction has calculated the sum */
    BEGIN TRANSACTION ISOLATION LEVEL serializable; 
    insert into counters(counter) values(10);
    COMMIT;

both transactions commit and the final table looks like:

1, 10, 1

which is possible if the first transaction committed first, and then the second transaction committed. but it is possible for another client to see the table as: [1], [1, 10], [1, 1, 10] which is a sequence of states which should not be possible. if you see [1], [1, 10] then you should see [1, 10, 11] as the last state. hence it violates external consistency.

Re: Apple open-sources FoundationDB

#447

Post by gregsfortytwo » Fri, Apr 27, 2018, 11:14 PM UTC

manigandham wrote: post #142

noahdesu wrote: post #107

Earlier quoted context omitted.

Not that this really has anything to do with FoundationDB, but why do you say that object storage is a poor substrate for file and block abstractions? There are many high-performance block and file systems built on object storage.

Block level is the lowest form of addressing bytes on devices. Filesystems are an abstraction on top of block devices. Object stores are an abstraction on filesystems. Emulating a low-level layer on a higher-level abstraction (which itself is using this hierarchy) will never match the speed, scale, or reliability of doing it correctly.

These abstraction statements just aren't true.

Yes, you can architect a storage system this way. But 1) Even if you do, many, many, many high-performance systems are "on top" of a filesystem but don't actually use the filesystem for anything except perhaps as a block allocator. Consider databases.

2) Many, many object stores do not abstract on top of filesystems. Modern RADOS, the distributed object store, stores its local data in a local object store called BlueStore. BlueStore speaks directly to the block device; there's no filesystem involved.

3) Even if you did store part of your distributed object store data on top of a local filesystem, that's not necessarily an issue. HDFS does this. (HDFS, despite the "FS" in its name, is an object store as most practitioners understand them.)

Re: Apple open-sources FoundationDB

#448

Post by gregsfortytwo » Fri, Apr 27, 2018, 11:21 PM UTC

catwell wrote: post #348

manigandham wrote: post #142

Earlier quoted context omitted.

Block level is the lowest form of addressing bytes on devices. Filesystems are an abstraction on top of block devices. Object stores are an abstraction on filesystems. Emulating a low-level layer on a higher-level abstraction (which itself is using this hierarchy) will never match the speed, scale, or reliability of doing it correctly.

> Block level is the lowest form of addressing bytes on devices. Filesystems are an abstraction on top of block devices. Object stores are an abstraction on filesystems. I don't agree with this, but I think you may be confused because "Object Storage" can mean several different things. "Object Store" in Ceph (as in RADOS - Reliable Autonomous Distributed Object Store) basically means key-value store. I typically say…

Just for future reference, RADOS is actually not very S3-like. It is an object store; it does map from object names to buckets of bytes. But unlike S3 and many similar object stores or key-value DBs, RADOS allows you to do file-like operations: you can append, write to random offsets in the object, overwrite pieces of it but not the whole object, etc. (That's all in addition to some stunningly-complex stuff like injecting custom code to do specific kinds of transactional read-writes on the OSD [storage node] itself.)

That's all key to RBD being useful, or indeed CephFS itself. There are systems that map a filesystem layer on top of S3, but they have trouble because there aren't good ways to overwrite random small pieces of an S3 object. With RADOS, there are! :)

Re: Apple open-sources FoundationDB

#449

Post by manigandham » Sat, Apr 28, 2018, 9:40 PM UTC

gregsfortytwo wrote: post #447

manigandham wrote: post #142

Earlier quoted context omitted.

Block level is the lowest form of addressing bytes on devices. Filesystems are an abstraction on top of block devices. Object stores are an abstraction on filesystems. Emulating a low-level layer on a higher-level abstraction (which itself is using this hierarchy) will never match the speed, scale, or reliability of doing it correctly.

These abstraction statements just aren't true. Yes, you can architect a storage system this way. But 1) Even if you do, many, many, many high-performance systems are "on top" of a filesystem but don't actually use the filesystem for anything except perhaps as a block allocator. Consider databases. 2) Many, many object stores do not abstract on top of filesystems. Modern RADOS, the distributed object store, stores its…

1) Yes, databases are just object stores with indexing and querying, which are an abstraction on filesystems, which are abstractions on block devices.

2) Rados is an object store, which is an abstraction on BlueStore (effectively a filesystem and replacement of FileStore), which is an abstraction on block devices.

3) HDFS is an object store, which is an abstraction on filesystems, which are an abstraction on block devices.

I'm not sure what you're point is because you just restated what I already said. They are abstraction, and they work just fine without any performance issues because that is the trade off of having an abstraction.

What I also said is that emulating low-level layers on a higher-level interface (like a block device on top of a database or object store) will never match the original block device. What is untrue about this?

Re: Apple open-sources FoundationDB

#450

Post by noahdesu » Sun, Apr 29, 2018, 3:05 PM UTC

manigandham wrote: post #449

gregsfortytwo wrote: post #447

Earlier quoted context omitted.

These abstraction statements just aren't true. Yes, you can architect a storage system this way. But 1) Even if you do, many, many, many high-performance systems are "on top" of a filesystem but don't actually use the filesystem for anything except perhaps as a block allocator. Consider databases. 2) Many, many object stores do not abstract on top of filesystems. Modern RADOS, the distributed object store, stores its…

1) Yes, databases are just object stores with indexing and querying, which are an abstraction on filesystems, which are abstractions on block devices. 2) Rados is an object store, which is an abstraction on BlueStore (effectively a filesystem and replacement of FileStore), which is an abstraction on block devices. 3) HDFS is an object store, which is an abstraction on filesystems, which are an abstraction on block de…

It appears that you want to make a very simple statement: "Abstractions tend to introduce overhead. If any software layers, an OS, or a network are added on top of a block device, I/O overhead will be introduced somewhere." As a conversational seed, I would wager most people would agree with that statement, in general.

One issue in this thread is that abstractions are concepts, not cpu instructions. In order to discuss overhead, one needs to reify the abstraction. For example, if you care about latency overhead, the block scheduler will definitely introduce overhead. But if you care about throughput, you probably /want/ abstractions like queues and schedulers.

> "What I also said is that emulating low-level layers on a higher-level interface (like a block device on top of a database or object store) will never match the original block device. What is untrue about this?"

Nothing is untrue about the sentiment of your statement. But from a practical standpoint, storage devices are useless pieces of junk without software. So to say abstractions slow down storage device while ignoring their utility feels arbitrary: why not talk about the length of the SATA cable, or the firmware in the disk controller? If the answer is that you just wanted to make the simple statement like the one I quoted at the start of this post then that's great, I think we are all in agreement. Otherwise, it's not clear what your point is and many of the supporting examples that you list are stated as fact, but are in reality either generally untrue, or very nuanced points, both of which tend to attract strong opinions :)