Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple open-sources FoundationDB

foundationdb.org

Post reply on HN
Print view View original

431–440 of 453 posts

Re: Apple open-sources FoundationDB

#431

Post by ddorian43 »

foodbaby wrote: post #397
zbentley wrote: post #290

Earlier quoted context omitted.

> but you had better have a good async RPC system! The microservices crew with their "our database is behind a REST/Thrift/gRPC/FizzBuzzWhatnot microservice" pattern is still catching up to the significance of this statement.

This might be a dumb question (from someone used to using blocking JBDC) but why is async RPC important in this case? Just trying to understand. And can gRPC not provide good async RPC?

Imagine single core single threaded design. You send 2 requests for 1 row each.

First request the row needs to be read from disk HDD. It takes 2ms.

Second request, the row is already in ram, it takes microseconds but still has to wait for the first request to finish.

Threads have overhead when having a lot of concurrency (thousands/millions requests/second).

For extreme async, see seastar-framework and scylladb design.

TLDR: high concurrency, low overhead etc.

Re: Apple open-sources FoundationDB

#432

Post by nunb »

srigi wrote: post #417
bsaul wrote: post #6

Earlier quoted context omitted.

Your talk was one of the best talk i've seen , and i keep mentionning it to people whenever they ask me about distributed systems, database and testing. i'm incredibly impatient to have a look at what the community is going to build on top of that very promising technology.

Can you share that talk here too?

link to the [talk] from the parent comment [by wwilson]

[talk] https://www.youtube.com/watch?v=4fFDFbi3toc

[by wwilson] https://news.ycombinator.com/item?id=16877401

Re: Apple open-sources FoundationDB

#433

Post by voidmain »

edwinyzh wrote: post #430
wwilson wrote: post #2

This is INCREDIBLE news! FoundationDB is the greatest piece of software I’ve ever worked on or used, and an amazing primitive for anybody who’s building distributed systems. The short version is that FDB is a massively scalable and fast transactional distributed database with some of the best testing and fault-tolerance on earth[1]. It’s in widespread production use at Apple and several other major companies. But the…

How scalable and feasible to implement a SQL Layer on top of SQLite's Virtual Table Mechanism ( https://www.sqlite.org/vtab.html ) which redirects the read/write of the record data from/to foundationdb?

Long before we acquired Akiban, I prototyped a sql layer using (now defunct) sqlite4, which used a K/V store abstraction as its storage layer. I would guess that a virtual table implementation would be similar: easy to get working, and it would work, but the performance is never going to be amazing.

To get great performance for SQL on FoundationDB you really want an asynchronous execution engine that can take full advantage of the ability to hide latency by submitting multiple queries to the K/V store in parallel. For example if you are doing a nested loop join of tables A and B you will be reading rows of B randomly based on the foreign keys in A, but you want to be requesting hundreds or thousands of them simultaneously, not one by one.

Even our SQL Layer (derived from Akiban) only got this mostly right - its execution engine was not originally designed to be asynchronous and we modified it to do pipelining which can get a good amount of parallelism but still leaves something on the table especially in small fast queries.

Re: Apple open-sources FoundationDB

#434

Post by djtriptych »

inteleng wrote: post #424
djtriptych wrote: post #51

I went to the same high school as the founders[1]. They were about the 2 best software engineers in a school with a LOT of very smart software engineers. Another pair founded Yext, which went public last year. I still consider that school the group with the highest concentration of raw brain power I've ever been a part of. I'm probably a 1% engineer, been hired by M$, FB, and Google. These guys were light years ahead…

Maybe it's just me, but the hubris in this comment seems a little excessive. They aforementioned SWEs make themselves multimillionaires /and/ have great jobs /and/ get praised by their peers, yet now everyone else has to bow down to them... over claims they were great programmers in high school? Is an appeal to your own experience the best way to make yourself seem relevant here? It's an incredible DB system, but it…

Did you bow down at your laptop?

Is it ok to admire great engineers on HN?

Is this comment relevant to anything but your own inferiority complex?

Can we get a decent definition of hubris in here?

Re: Apple open-sources FoundationDB

#435

Post by mogui »

voidmain wrote: post #271
ddorian43 wrote: post #258

Earlier quoted context omitted.

Wouldn't layers be hard to be built on the server (since you have to also change the client) and slow to be built as a layer (since it will be another separate service) ?

I'm not sure what you are asking, but depending on their individual performance and security needs layers are usually either (a) libraries embedded into their clients, (b) services colocated with their clients, (c) services running in a separate tier, or (d) services co-located with fdbservers. In any of these cases they use the FoundationDB client to communicate with FoundationDB.

In case (c) or (d) how can a layer leverage the distributed facilities that FDB gives? I mean if I have clients that connect to a "layer service" that is the one who talks to FDB, I have to manage "layer service" scalabily, fault tolerance etc... by myself.

Re: Apple open-sources FoundationDB

#436

Post by edwinyzh »

voidmain wrote: post #433
edwinyzh wrote: post #430

Earlier quoted context omitted.

How scalable and feasible to implement a SQL Layer on top of SQLite's Virtual Table Mechanism ( https://www.sqlite.org/vtab.html ) which redirects the read/write of the record data from/to foundationdb?

Long before we acquired Akiban, I prototyped a sql layer using (now defunct) sqlite4, which used a K/V store abstraction as its storage layer. I would guess that a virtual table implementation would be similar: easy to get working, and it would work, but the performance is never going to be amazing. To get great performance for SQL on FoundationDB you really want an asynchronous execution engine that can take full ad…

@voidmain, Thank you, it's very insightful and clear! I mean, I can see the disadvantage if such SQL layer is implemented directly through SQLite's virtual tables.

Re: Apple open-sources FoundationDB

#437

Post by inteleng »

djtriptych wrote: post #434
inteleng wrote: post #424

Earlier quoted context omitted.

Maybe it's just me, but the hubris in this comment seems a little excessive. They aforementioned SWEs make themselves multimillionaires /and/ have great jobs /and/ get praised by their peers, yet now everyone else has to bow down to them... over claims they were great programmers in high school? Is an appeal to your own experience the best way to make yourself seem relevant here? It's an incredible DB system, but it…

Did you bow down at your laptop? Is it ok to admire great engineers on HN? Is this comment relevant to anything but your own inferiority complex? Can we get a decent definition of hubris in here?

I don't bow down to my laptop, but I respect what went into making it.

It's great to admire excellent engineers; aspiring to be as skilled as someone at a task can be very motivating. Worshipping them is another thing.

You're right about the inferiority complex - I know I'm a relatively bad SW engineer, but that's mostly related to how new it is to me. I expect and want to improve.

Hubris is defined as excessive pride or self-confidence. I would say that bragging that you're "probably a 1% engineer," and that you've been hired by three of the largest SW companies out there qualifies as hubris. Maybe it's just me, but I don't think a 1% engineer would publicly boast about being one and then actually use the 'M$' in a non-farcical manner.

Shitting on 99% of the SWE population to make yourself look good, then shitting on yourself to make another person look even better doesn't really work. There's a reason humanCmp() is a little more complex than strCmp().

BTW, being hired by a large company doesn't mean you're all that. Plenty of idiots get hired by Oracle.

Re: Apple open-sources FoundationDB

#438

Post by cakoose »

evanweaver wrote: post #164
isoos wrote: post #61

How does it compare to CockroachDB or TiDB?

It's closest to TiDB's key-value layer; a building block for more complex systems. More traditional, monolithic databases like CockroachDB (SQL) or FaunaDB (NoSQL) trade off extensibility for the benefits in performance and operations that come from very tight coupling. In my understanding, FoundationDB's transaction management is closest to FaunaDB's; read/write sets are linearized in memory in preprocessing nodes a…

> monolithic databases like CockroachDB (SQL) or FaunaDB (NoSQL) trade off extensibility

The CockroachDB documentation says their SQL implementation is layered on top of their K/V interface: https://www.cockroachlabs.com/docs/stable/architecture/overv...

This would make it similar to TiDB/TiKV and FoundationDB.

Here's and old article from VoltDB insisting that layering SQL on pure K/V forfeits too much performance: https://www.voltdb.com/blog/2015/04/01/foundationdbs-lesson-...

Edit: And a response: https://news.ycombinator.com/item?id=15505194

Re: Apple open-sources FoundationDB

#439

Post by zbentley »

foodbaby wrote: post #397
zbentley wrote: post #290

Earlier quoted context omitted.

> but you had better have a good async RPC system! The microservices crew with their "our database is behind a REST/Thrift/gRPC/FizzBuzzWhatnot microservice" pattern is still catching up to the significance of this statement.

This might be a dumb question (from someone used to using blocking JBDC) but why is async RPC important in this case? Just trying to understand. And can gRPC not provide good async RPC?

I was referring to the trend of splitting up applications into highly distributed collections of services without addressing the fact that every point where they communicate over the network is a potential point of pathological failure (from blocking to duplicate-delivery etc). This tendency replaces highly reliable network protocols (i.e. the one you use to talk to your RDBMS) with ad hoc and frequently technically shoddy communication patterns, with minimal consideration for how it might fail in complex, distributed ways. While not always done wrong, a lot of microservice-ification efforts are quite hubristic in this area, and suffer for it over the long term.

Re: Apple open-sources FoundationDB

#440

Post by voidmain »

mogui wrote: post #435
voidmain wrote: post #271

Earlier quoted context omitted.

I'm not sure what you are asking, but depending on their individual performance and security needs layers are usually either (a) libraries embedded into their clients, (b) services colocated with their clients, (c) services running in a separate tier, or (d) services co-located with fdbservers. In any of these cases they use the FoundationDB client to communicate with FoundationDB.

In case (c) or (d) how can a layer leverage the distributed facilities that FDB gives? I mean if I have clients that connect to a "layer service" that is the one who talks to FDB, I have to manage "layer service" scalabily, fault tolerance etc... by myself.

Yes, and that's the main advantage of choosing (a) or (b). But it's not quite as hard as it sounds; since all your state is safely in fdb you "just" have to worry about load balancing a stateless service.
Post reply on HN