Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple open-sources FoundationDB

foundationdb.org

Post reply on HN
Print view View original

401–410 of 453 posts

Re: Apple open-sources FoundationDB

#401

Post by gazarsgo »

burntsushi wrote: post #357
fizx wrote: post #211

Earlier quoted context omitted.

It'd take a low number of hours to wire up FoundationDB as a Lucene filesystem (Directory) implementation. Shared filesystem with a local RAM cache has been practical for a while in Lucene, and was briefly supported then deprecated in Elasticsearch. I've used Lucene on top of HDFS and S3 quite nicely. If you have a reason to use FoundationDB over HDFS, NFS, S3, etc, then this will work well. Doing a Lucene+DB impleme…

> I've used Lucene on top of HDFS and S3 quite nicely. Out of curiosity, what led you to do this? And what does it do better/worse/differently than out-of-the-box things like Elasticsearch or SOLR?

HDFS is supported by Solr+Lucene, but rather than my own poor paraphrasing, see what you think of this writeup: https://engineering.linkedin.com/search/did-you-mean-galene

Re: Apple open-sources FoundationDB

#403

Post by agibsonccc »

jamesblonde wrote: post #326
notacoward wrote: post #299

Earlier quoted context omitted.

> best known metadata storage engine for a distributed filesystem, hopsfs Not even close. I don't even see anything I'd call a filesystem mentioned on your web page. I missed FAST this year, and apparently you had a paper about using Hops as a building block for a non-POSIX filesystem - i.e. not a filesystem in my and many others' opinion - but it's not clear whether it has ever even been used in production anywhere…

Yes, it's used in production. There's a company commercializing it: www.logicalclocks.com

I'm not convinced a paper with a website and some proof of concepts would be considered the "best". You're throwing around a bunch of components in to a distro calling yourselves everything from "deep learning" to a file system. It's not clear what you guys are even trying to do here.

Re: Apple open-sources FoundationDB

#404

Post by dtheodor »

Since becoming open source and the system's architecture is in the open, I wonder how this criticism of FoundationDB from a VoltDB architect https://www.voltdb.com/blog/2015/04/01/foundationdbs-lesson-... fairs against the knowledge that has now become available. (To summarize, the author argues that building an SQL layer on top of an ordered key-value store is suboptimal)

Re: Apple open-sources FoundationDB

#405

Post by polskibus »

voidmain wrote: post #391
polskibus wrote: post #373

Is foundationdb capable of providing a linearizable data store? As I remember from Martin Kleppman's books Serializable Snapshot Isolation is not linearizable because the snapshot does not include writes more recent than itself.

Yes. Linearizable means both serializable and externally consistent (or is sometimes used as just a synonym for the latter), and FDB has these properties with respect to transactions.

That would mean that SSI is not used to provide Serializable isolation. level. If so, what is used instead? 2 phase locking? I thought it's not very scalable ?

Re: Apple open-sources FoundationDB

#406

Post by edwinyzh »

wwilson wrote: post #60
thefounder wrote: post #53

Earlier quoted context omitted.

I'm not familiar with FDB but what you say sounds almost too good to be true. Can I use it to implement the Google Datastore api? I'm trying for years to find a suitable backend so that I can leave the Google land. Everything I tried either required a schema or lacked transactions or key namespaces.

As an existence proof: before the acquisition we built an ANSI SQL database and a wire-compatible clone of the MongoDB API. I see no reason you wouldn't be able to implement Datastore. In fact here's a public source claiming that Firestore (which I believe is its successor) is implemented on top of Spanner: https://www.theregister.co.uk/2017/10/04/google_backs_up_fir...

I hope this question doesn't feel too dump, but is it possible to implement a SQL Layer using SQLite's Virtual Table mechanism and leverage all foundationdb's features?

Re: Apple open-sources FoundationDB

#407

Post by burntsushi »

gazarsgo wrote: post #401
burntsushi wrote: post #357

Earlier quoted context omitted.

> I've used Lucene on top of HDFS and S3 quite nicely. Out of curiosity, what led you to do this? And what does it do better/worse/differently than out-of-the-box things like Elasticsearch or SOLR?

HDFS is supported by Solr+Lucene, but rather than my own poor paraphrasing, see what you think of this writeup: https://engineering.linkedin.com/search/did-you-mean-galene

Ah, excellent! Thanks. That answers my question. I also found the idea of early termination via static rank very intriguing.

Re: Apple open-sources FoundationDB

#408

Post by mst »

aeorgnoieang wrote: post #376
chx wrote: post #159

Earlier quoted context omitted.

> That API is gnarly. Out of the many MongoDB criticisms, this one is valid. This https://www.linkedin.com/pulse/mongodb-frankenstein-monster-... article is quite right about it (note: endorsing the article does not mean I endorse its author by far). Other than that, they totally did a fake it until you make it with MongoDB 3.4 passing Jepsen a year ago and MongoDB BI 2.0 containing their own SQL engine instead of wr…

What specifically are you trying to avoid endorsing about the author of the LinkedIn post to which you linked? I couldn't find anything from a cursory web search.

He runs lambdaconf, and refused to disinvite a speaker who many people felt shouldn't be permitted to speak because of his historical non-technical writings.

(I've tried to keep the above as dry as possible to avoid dragging the arguments around this situation into this thread - and I suspect the phrasing of the previous comment was also intended to try and avoid that, so let's see if we can keep it that way, please)

Re: Apple open-sources FoundationDB

#409

Post by jahhein »

wwilson wrote: post #2

This is INCREDIBLE news! FoundationDB is the greatest piece of software I’ve ever worked on or used, and an amazing primitive for anybody who’s building distributed systems. The short version is that FDB is a massively scalable and fast transactional distributed database with some of the best testing and fault-tolerance on earth[1]. It’s in widespread production use at Apple and several other major companies. But the…

That presentation was really good! Well explained on the simulations. If one wanted to get into this exciting event and create something with FoundationDB but no database experience (I do know many programming languages) where would I start? If anyone could point me in the direction, I'd greatly appreciate it.

Re: Apple open-sources FoundationDB

#410

Post by voidmain »

polskibus wrote: post #405
voidmain wrote: post #391

Earlier quoted context omitted.

Yes. Linearizable means both serializable and externally consistent (or is sometimes used as just a synonym for the latter), and FDB has these properties with respect to transactions.

That would mean that SSI is not used to provide Serializable isolation. level. If so, what is used instead? 2 phase locking? I thought it's not very scalable ?

I explain the basics of our concurrency control here: https://news.ycombinator.com/item?id=16877950

I guess textbook SSI is willing to "reorder" conflicting transactions if the result is still serializable, which could violate external consistency if you don't have any other bounds on the order. In the language of SSI, fdb simply aborts the later of any pair of read/write transactions with an rw-conflict, in accordance with a fixed ordering which is externally consistent.

I guess it could also be that your book uses an idiosyncratic definition of linearizable, like trying to apply it to individual operations within transactions, which might rule out any optimistic concurrency method. It might just be better to delete this word from your vocabulary in the database field because there is no wide agreement on what it means. The first two hits on Google for me are Wikipedia and Peter Bailis, and they give clearly conflicting definitions, though I think fdb satisfies both!

Post reply on HN