Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FoundationDB Summit Program Announced

foundationdb.org

Post reply on HN
Print view View original

11–20 of 24 posts

Re: FoundationDB Summit Program Announced

#11

Post by devj »

monstrado wrote: post #7
dominotw wrote: post #6

Earlier quoted context omitted.

Curious what ur usecase is and what makes foundationDB particularly suited for it.

Real-time aggregations over a stream of data where we may have multiple servers writing a partial aggregation to the same row. With FDB I can safely read data, merge it with my in-memory copy, and then write the final result back. That's only for our complex aggregations, such as HyperLogLog and T-Digests. For the easier things like COUNT I can just use the ADD mutation. For SUM of doubles, I can use APPEND_IF_FITS t…

Wow.. Which client language are you using?

Re: FoundationDB Summit Program Announced

#12

Post by mping »

monstrado wrote: post #7
dominotw wrote: post #6

Earlier quoted context omitted.

Curious what ur usecase is and what makes foundationDB particularly suited for it.

Real-time aggregations over a stream of data where we may have multiple servers writing a partial aggregation to the same row. With FDB I can safely read data, merge it with my in-memory copy, and then write the final result back. That's only for our complex aggregations, such as HyperLogLog and T-Digests. For the easier things like COUNT I can just use the ADD mutation. For SUM of doubles, I can use APPEND_IF_FITS t…

That's pretty cool! You implemented HLL and TDigest on top of FDB? Or are you storing some kind of blob and computing server-side? I did something similar for Hadoop and MonetDB a long time ago

Re: FoundationDB Summit Program Announced

#13

Post by asien »

innagadadavida wrote: post #9

Apple internally uses Cassandra, HBase, Riak, Hadoop/Impala, Oracle, Siri Search kv store, memcache, redis, MySQL, Postgres etc etc. Each of these handle >100 TB in aggregate. Considering these applications won’t be ported to FDB, why not develop a translation later. This will also drive adoption of FDB. Having smart people work on cool things is not sufficient, you also need them to be working on solving high impact…

>Considering these applications won’t be ported to FDB, why not develop a translation later. This will also drive adoption of FDB.

Writing a translation layer would be nice, "drop in" replacement can be an overkill to drive growth.

That said the three biggest factors for adoption in my opinion are developer experience, tooling and hosting.

If some FoundationDB enthusiasts made an elastic hosting service and some dedicated tooling it would be help massively to compete with others NoSQL Vendors.

Re: FoundationDB Summit Program Announced

#14

Post by CodesInChaos »

john92 wrote: post #10

Considering one database to rule all is old..where does this stand? https://www.allthingsdistributed.com/2018/06/purpose-built-d...

FoundationDB is a key-value store which supports transactions and scans over key ranges. Then on top of that layer, you can add higher level abstractions, like relational, document, graph databases. So it's similar to the storage engine concept many database servers use.

Re: FoundationDB Summit Program Announced

#17

Post by solarengineer »

asien wrote: post #13
innagadadavida wrote: post #9

Apple internally uses Cassandra, HBase, Riak, Hadoop/Impala, Oracle, Siri Search kv store, memcache, redis, MySQL, Postgres etc etc. Each of these handle >100 TB in aggregate. Considering these applications won’t be ported to FDB, why not develop a translation later. This will also drive adoption of FDB. Having smart people work on cool things is not sufficient, you also need them to be working on solving high impact…

>Considering these applications won’t be ported to FDB, why not develop a translation later. This will also drive adoption of FDB. Writing a translation layer would be nice, "drop in" replacement can be an overkill to drive growth. That said the three biggest factors for adoption in my opinion are developer experience, tooling and hosting. If some FoundationDB enthusiasts made an elastic hosting service and some dedi…

What dedicated tooling would you like to see? What would an elastic hosting experience feel like to you?

Re: FoundationDB Summit Program Announced

#18

Post by monstrado »

devj wrote: post #11
monstrado wrote: post #7

Earlier quoted context omitted.

Real-time aggregations over a stream of data where we may have multiple servers writing a partial aggregation to the same row. With FDB I can safely read data, merge it with my in-memory copy, and then write the final result back. That's only for our complex aggregations, such as HyperLogLog and T-Digests. For the easier things like COUNT I can just use the ADD mutation. For SUM of doubles, I can use APPEND_IF_FITS t…

Wow.. Which client language are you using?

Interfacing with the Java client using Scala

Re: FoundationDB Summit Program Announced

#19

Post by monstrado »

mping wrote: post #12
monstrado wrote: post #7

Earlier quoted context omitted.

Real-time aggregations over a stream of data where we may have multiple servers writing a partial aggregation to the same row. With FDB I can safely read data, merge it with my in-memory copy, and then write the final result back. That's only for our complex aggregations, such as HyperLogLog and T-Digests. For the easier things like COUNT I can just use the ADD mutation. For SUM of doubles, I can use APPEND_IF_FITS t…

That's pretty cool! You implemented HLL and TDigest on top of FDB? Or are you storing some kind of blob and computing server-side? I did something similar for Hadoop and MonetDB a long time ago

Both the HLL (Algebird) and TDigest implementations we're using have a simple way to serialize a compressed representation. So basically just reading the row, merging the value currently stored, and writing the merged value back.

Depending on how many times you will write to the row, you could avoid having to do a merge on write by using APPEND_IF_FITS and just merging the byte arrays when you read.

It's nice that FDB gives you so much low level flexibility, you can do whatever you feel fits your use case.

Re: FoundationDB Summit Program Announced

#20

Post by scaleout1 »

monstrado wrote: post #19
mping wrote: post #12

Earlier quoted context omitted.

That's pretty cool! You implemented HLL and TDigest on top of FDB? Or are you storing some kind of blob and computing server-side? I did something similar for Hadoop and MonetDB a long time ago

Both the HLL (Algebird) and TDigest implementations we're using have a simple way to serialize a compressed representation. So basically just reading the row, merging the value currently stored, and writing the merged value back. Depending on how many times you will write to the row, you could avoid having to do a merge on write by using APPEND_IF_FITS and just merging the byte arrays when you read. It's nice that FD…

Hey man, thats pretty cool and we do exactly the same using Cassandra instead of FDB. Since Cassandra doesnt support transaction at high volume (100K tps) we do a shuffle so that all the same key do read/modify/write from the same machine. It seems like with FDB you can get away with it as it supports transactions? My question to you is what is the volume your system is operating at? Also how does it work for skews? Lets say you need to update HLL for a key that is heavily skewed, does your FDB transaction unwind fast enough not to slow down the whole system?
Post reply on HN