Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FoundationDB 6.0 released, featuring multi-region support

foundationdb.org

Post reply on HN
Print view View original

21–30 of 72 posts

Re: FoundationDB 6.0 released, featuring multi-region support

#21

Post by zzzcpan »

dividuum wrote: post #18
ryanworl wrote: post #4

To anyone who is on the fence about putting FoundationDB into production (or at least evaluating it for their use cases), what is the number one thing you think is missing or you're worried about? i.e. - a SQL interface, - pre-packaged data structure libraries, - monitoring, - limitations of FoundationDB itself, - etc. I'm working on a talk for the upcoming FoundationDB Summit and I'd love to address some real-world…

One of my constant pain point with all distributed data stores is that it's really hard to find out how they behave if something breaks. Be it the network, local storage and so on. How do I find out what's wrong? Are there guides on how to fix a problem? What happens if I lost more nodes than required to automatically recover? How does backup and restore work? Any estimates on how long a restore will take? Are there…

Yeah, operations is something nobody talks about.

Re: FoundationDB 6.0 released, featuring multi-region support

#22

Post by amirouche »

htfy96 wrote: post #13
DLA wrote: post #10

Earlier quoted context omitted.

+1 for a SQL layer/interface. Would massively increase utility of FDB.

Haven't tested yet, but I was wondering if it's possible to use TiDB as SQL layer. [0] [0]: https://github.com/apple/foundationdb/issues/198

https://forums.foundationdb.org/t/sql-layer-in-foundationdb/...

Re: FoundationDB 6.0 released, featuring multi-region support

#23

Post by amirouche »

ryanworl wrote: post #4

To anyone who is on the fence about putting FoundationDB into production (or at least evaluating it for their use cases), what is the number one thing you think is missing or you're worried about? i.e. - a SQL interface, - pre-packaged data structure libraries, - monitoring, - limitations of FoundationDB itself, - etc. I'm working on a talk for the upcoming FoundationDB Summit and I'd love to address some real-world…

Deployement strategies guide from single node cluster to single datacenter to multi region. And how version upgrades happens

Re: FoundationDB 6.0 released, featuring multi-region support

#24

Post by lykr0n »

ryanworl wrote: post #4

To anyone who is on the fence about putting FoundationDB into production (or at least evaluating it for their use cases), what is the number one thing you think is missing or you're worried about? i.e. - a SQL interface, - pre-packaged data structure libraries, - monitoring, - limitations of FoundationDB itself, - etc. I'm working on a talk for the upcoming FoundationDB Summit and I'd love to address some real-world…

My blocker is that the documentation is confusing and not very straight forward. The tutorials are not very verbose and the code (at least for python) is not written in a way that is easy for someone to understand.

I'd be very interested in usage for time-series, but it looks like to get a working example up I'd need to fully parse the documentation and tutorials in order to do it as there is no "Let's build a Timeseries Database on FoundationDB from scratch"

Re: FoundationDB 6.0 released, featuring multi-region support

#25

Post by ex3ndr »

ryanworl wrote: post #4

To anyone who is on the fence about putting FoundationDB into production (or at least evaluating it for their use cases), what is the number one thing you think is missing or you're worried about? i.e. - a SQL interface, - pre-packaged data structure libraries, - monitoring, - limitations of FoundationDB itself, - etc. I'm working on a talk for the upcoming FoundationDB Summit and I'd love to address some real-world…

We are actually spend two weeks rewriting our code from SQL to FoundationDB (we have built our own simple ORM).

One of the huge problems is a lack of consistency constraints. Since during development data make became broken in various ways (and we have done several times). And right now there are no way to implement simplest constraints on database level.

Re: FoundationDB 6.0 released, featuring multi-region support

#26

Post by ryanworl »

ex3ndr wrote: post #25
ryanworl wrote: post #4

To anyone who is on the fence about putting FoundationDB into production (or at least evaluating it for their use cases), what is the number one thing you think is missing or you're worried about? i.e. - a SQL interface, - pre-packaged data structure libraries, - monitoring, - limitations of FoundationDB itself, - etc. I'm working on a talk for the upcoming FoundationDB Summit and I'd love to address some real-world…

We are actually spend two weeks rewriting our code from SQL to FoundationDB (we have built our own simple ORM). One of the huge problems is a lack of consistency constraints. Since during development data make became broken in various ways (and we have done several times). And right now there are no way to implement simplest constraints on database level.

Could you give an example of what you mean? This is very relevant for my talk.

Re: FoundationDB 6.0 released, featuring multi-region support

#27

Post by riku_iki »

tensor wrote: post #8
ryanworl wrote: post #4

To anyone who is on the fence about putting FoundationDB into production (or at least evaluating it for their use cases), what is the number one thing you think is missing or you're worried about? i.e. - a SQL interface, - pre-packaged data structure libraries, - monitoring, - limitations of FoundationDB itself, - etc. I'm working on a talk for the upcoming FoundationDB Summit and I'd love to address some real-world…

We use Postgresql heavily and it's not really a viable replacement. Even if there were an SQL layer, I doubt it would be fully featured enough for our needs. We may evaluate FDB as a distributed cache, but haven't had the time.

Curious if it is possible to query fdb as fdw from psql..

Re: FoundationDB 6.0 released, featuring multi-region support

#28

Post by ex3ndr »

the_duke wrote: post #9

So, FoundationDB is a pretty low level distributed key-value store with transactions. Most applications will need something higher level, like a SQL or document db frontend which could be built on top. I'm curios what people have started using FoundationDB for. Any interesting stories to share?

We have moved to FDB for our messaging platform. We had several options:

a) Rewrite SQL code. In our case we are using Node.JS and all SQL libraries are very very slow. Even replacing one with another is enormous work. b) Rewrite to a new language. It was also an option since querying Postgres can take 1ms, but parsing response can easily take 100ms+. That trashed our event loop and causes awful latency. c) Rewrite to high performance NoSQL database.

We picked a last one. In context of a Node.js we were able to write really thin layer on top of FDB that works super-fast and in a way we needed.

In my previous startup we eventually ditched all SQL from our codebase too since SQL databases is just too slow for low latency messaging apps. There are no simple way to shard data, there are always random locks around your database (which blocks connections). Locks are really hard to debug sometimes. How to scale single SQL server? All of this is doable, but in FDB it was basically free.

We migrated to FDB and got almost x100 improvement in latency/performance. And unlike SQL-code that was very carefully crafted we can do nasty things. Like - "hey, let's just pull this key every 100ms and check for a new value" or "hey, let's do it on 10s of instances at the same time?". In this situations Postgres started to consume all available CPU. You can easily creep out SQL with a single instance of your app. We haven't managed this to do with FDB for 1/2 of the cost. We are often in situation when someone commit something with a bug and, for example, started to pull data every millisecond in N^2 streams where N is number of online users. In this situations we just can't see any impact at all on our platform. Just spikes in monitoring.

FDB is wonderful thing - it allows you to forget about optimizing performance of your queries, forget about managing backups and replication. It just works!

Re: FoundationDB 6.0 released, featuring multi-region support

#30

Post by ex3ndr »

ryanworl wrote: post #26
ex3ndr wrote: post #25

Earlier quoted context omitted.

We are actually spend two weeks rewriting our code from SQL to FoundationDB (we have built our own simple ORM). One of the huge problems is a lack of consistency constraints. Since during development data make became broken in various ways (and we have done several times). And right now there are no way to implement simplest constraints on database level.

Could you give an example of what you mean? This is very relevant for my talk.

I mean we have built a nice ORM for FoundationDB, but eventually we found several bugs in it and we just didn't write to our indexes correctly or put several indexes in the same subspace. Then we started to migrate our data (we were so crazy that moved large chunk of our production data already!) and we found that our migrators are also have problems. Basically our data was not consistent at all.

There are almost no easy way to ensure that everything is OK.

Also I have fear that we probably deleted something and there are no way to prohibit modification of certain keys.

Post reply on HN