Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FoundationDB: A distributed, unbundled, transactional key value store [pdf]

foundationdb.org

Post reply on HN
Print view View original

61–70 of 103 posts

Re: FoundationDB: A distributed, unbundled, transactional key value store [pdf]

#61

Post by Meai »

Personally I don't understand how you can call a database robust if it can't scale down nodes after you scaled them up once. What am I supposed to do if I ever deploy to 50 nodes and then it turns out that I only need 5. Shut the business down? Pay to run database servers forever that I don't even need anymore? Also the database configuration has a lot of gotchas and is very opaque. You might be waiting for 30sec for your CLI to connect to your localhost cluster of two processes and you have no idea what is happening or why it is taking that long. It just never felt so safe and robust as people claim it to be. I don't know, these were just my findings on the brief tests I did with it.

Also you better get familiar with a whole bunch of hidden "knobs" that are apparently configurable and very important somewhere and then get printed out into xml logs but of course there is no log viewer so you have to write your own. Maybe this isn't a problem for large companies but I'm providing feedback as a single user here.

I also don't understand how people can praise the c++ DSL. They should rewrite that into standard c++ coroutines as soon as possible so their entire build and dev environment isn't so hard to understand. As a user of open source software I generally like to be able to debug through the projects I use and figure out problems I have. It's much harder when a project uses their own custom language. I certainly tried to set it all up correctly but there always seemed to be some problems in regards to Intellisense within the IDE.

Re: FoundationDB: A distributed, unbundled, transactional key value store [pdf]

#62

Post by __float »

Meai wrote: post #61

Personally I don't understand how you can call a database robust if it can't scale down nodes after you scaled them up once. What am I supposed to do if I ever deploy to 50 nodes and then it turns out that I only need 5. Shut the business down? Pay to run database servers forever that I don't even need anymore? Also the database configuration has a lot of gotchas and is very opaque. You might be waiting for 30sec for…

https://apple.github.io/foundationdb/administration.html#rem...

What is this process describing, and how does it differ from what you were trying to do?

Re: FoundationDB: A distributed, unbundled, transactional key value store [pdf]

#63

Post by Meai »

__float wrote: post #62
Meai wrote: post #61

Personally I don't understand how you can call a database robust if it can't scale down nodes after you scaled them up once. What am I supposed to do if I ever deploy to 50 nodes and then it turns out that I only need 5. Shut the business down? Pay to run database servers forever that I don't even need anymore? Also the database configuration has a lot of gotchas and is very opaque. You might be waiting for 30sec for…

https://apple.github.io/foundationdb/administration.html#rem... What is this process describing, and how does it differ from what you were trying to do?

They are using the term "machine" and "process" in multiple conflicting meanings I think. I mean maybe they improved this since I used it 1.5 years ago but I kind of doubt it.

If I remember correctly there is a definite problem if you remove one of the processes that end up guaranteeing your configured redundancy mode. So then suddenly your entire cluster is inoperable. Yes really, I think it doesn't even properly respond to cli commands anymore and shows nodes as simply missing. Oh and suddenly those long wait times for your cli commands are really starting to bother you... I don't really want to talk more about it because it's been a while but I just wanted to make the point that the user experience was quite bad.

Re: FoundationDB: A distributed, unbundled, transactional key value store [pdf]

#66

Post by qaq »

eloff wrote: post #52
cwp wrote: post #49

Earlier quoted context omitted.

Yeah, but there are quite a few efforts out there to extend PG into a distributed DB of one flavor or another. Some examples are YugabyteDB, CockroachDB, Aurora and Citus. It's a reasonable approach, but it's also reasonable to come at it from the other direction - build a SQL engine on top of a solid distributed key-value store. Contrafactuals are always dicey, but FDB vanishing behind the Apple wall of silence sure…

> Some examples are YugabyteDB, CockroachDB, Aurora and Citus. Of those the first two are not PG, they just share the wire protocol and try to be compatible at a SQL level. Aurora is not really distributed, it's replicated for availability and durability six ways at the block storage level. Citus is distributed as I understand it though. > but it's also reasonable to come at it from the other direction - build a SQL…

Yugobyte is slightly altered PG "Frontend" with custom storage backend

Re: FoundationDB: A distributed, unbundled, transactional key value store [pdf]

#67

Post by davgoldin »

Only great things to say about FoundationDB. We've been using it for about a year now. Got a tiny, live cluster of 35+ commodity machines (started with 3 a year ago), about 5TB capacity and growing. Been removing and adding servers (on live cluster) without a glitch. We've got another 100TB cluster in testing now. Of all the things, we're actually using it as a distributed file system.

We've tried Ceph, GlusterFS, HDFS, MinIO and some others, and eventually decided on a custom FDB solution. It's a breeze to setup, and seems to eclipse others in performance [0] and reliability - Kyle (aphyr) the author of Jepsen series on distributed systems correctness, said: "haven't tested foundation in part because their testing appears to be waaaay more rigorous than mine." [1]

The way we use FDB, if anyone is interested, is we simply split files into small chunks (per FDB data design recommendations), and store all file's & folder's meta data in FDB such as byte count, create/access/write times, permissions, and a lot more. Folders are handled by the builtin Directory layer [2].

[0] https://apple.github.io/foundationdb/performance.html

[1] https://web.archive.org/web/20150312112552/http://blog.found...

[2] https://forums.foundationdb.org/t/whats-the-purpose-of-the-d...

Re: FoundationDB: A distributed, unbundled, transactional key value store [pdf]

#68

Post by tpetry »

davgoldin wrote: post #67

Only great things to say about FoundationDB. We've been using it for about a year now. Got a tiny, live cluster of 35+ commodity machines (started with 3 a year ago), about 5TB capacity and growing. Been removing and adding servers (on live cluster) without a glitch. We've got another 100TB cluster in testing now. Of all the things, we're actually using it as a distributed file system. We've tried Ceph, GlusterFS, HD…

That's a really interesting solution. Can you tell us more about it? Operating distributed blob storage systems is kind of fragile with every software i have yet tried.

Re: FoundationDB: A distributed, unbundled, transactional key value store [pdf]

#69

Post by jFriedensreich »

vosper wrote: post #59
jng wrote: post #54

Earlier quoted context omitted.

Lucene is Java, right? There should be space for a native implementation, like ScyllaDB is doing to Cassandra (and DynamoDB, though the gap is not of the same shape in the last case). Or am I missing something? I used ElasticSearch and run one cluster in production in the past and found it horrible. Maybe I'm missing something, since they're so successful even as a public company...

There’s a lot to Lucene, and there’s also a lot to ElasticSearch. And I think they’re fairly tightly coupled. But I do think that a well-funded and skilled startup team could take a run at ES. They’re a monopoly in their niche. There has to be money in disrupting them.

im sure its possible in theory, but without a permissive license it would be not relevant to most applications that are interesting. now try convincing a venture backed startup to build a lucene alternative and license it permissively after seeing what happend with amazon and es...

Re: FoundationDB: A distributed, unbundled, transactional key value store [pdf]

#70

Post by jFriedensreich »

jng wrote: post #54
jFriedensreich wrote: post #47

Earlier quoted context omitted.

its pretty hard to catch up with lucene, there is just so much work, features and brainpower in there at this point. as many features of foundationdb such as the transaction guarantees and reliability are not super important for fulltext search i cannot imagine any company even apple or ibm being able to justify that gigantic investment, instead im sure nearly any soluion willcontinue to use lucene under the hood for…

Lucene is Java, right? There should be space for a native implementation, like ScyllaDB is doing to Cassandra (and DynamoDB, though the gap is not of the same shape in the last case). Or am I missing something? I used ElasticSearch and run one cluster in production in the past and found it horrible. Maybe I'm missing something, since they're so successful even as a public company...

yes its java and also yes there is definitely a space for a native implementation, its not that i don't want that to happen, on the contrary. but the reality is that its really hard to do and i dont see only getting rid of java to currently motivate a relevant player or large enough dev community.
Post reply on HN