Random inside joke I overheard about ten years ago:
"It's called BigTable, not FastTable or AvailableTable!"
...It's probably a bad idea to evaluate 2019's BigTable based on the joke, but my puerile mind still find it amusing. :)
Joy and Pain of Using Google BigTable
11–20 of 21 posts
Re: Joy and Pain of Using Google BigTable
#12Reading the article the following quote got my attention "you should always keep things simple even if your tools allow for more complex patterns".
I follow the "it is perfect when you don't need to remove anything else" rule in most systems/processes/functions/tasks in life (not only IT systems). I am happy to see in this cluttered space called IT there are many more like-minded people who see that too much is TOO much.
Re: Joy and Pain of Using Google BigTable
#13Reading the article the following quote got my attention "you should always keep things simple even if your tools allow for more complex patterns". I follow the "it is perfect when you don't need to remove anything else" rule in most systems/processes/functions/tasks in life (not only IT systems). I am happy to see in this cluttered space called IT there are many more like-minded people who see that too much is TOO m…
Obligatory, "Does this function bring you joy?"
Re: Joy and Pain of Using Google BigTable
#14Worth noting his original reason for moving away from DynamoDB is outdated. DynamoDB added an “adaptive capacity” feature to handle hot partitions.[1] [1] https://aws.amazon.com/blogs/database/how-amazon-dynamodb-ad...
It's still expensive though, right?
Re: Joy and Pain of Using Google BigTable
#15Reading the article the following quote got my attention "you should always keep things simple even if your tools allow for more complex patterns". I follow the "it is perfect when you don't need to remove anything else" rule in most systems/processes/functions/tasks in life (not only IT systems). I am happy to see in this cluttered space called IT there are many more like-minded people who see that too much is TOO m…
Obligatory, "Does this function bring you joy?"
It's a beautiful thing that this comment exists :-D
Re: Joy and Pain of Using Google BigTable
#16Worth noting his original reason for moving away from DynamoDB is outdated. DynamoDB added an “adaptive capacity” feature to handle hot partitions.[1] [1] https://aws.amazon.com/blogs/database/how-amazon-dynamodb-ad...
It's still expensive though, right?
Compared to what? You'll have to be more specific.
Definitely adaptive capacity targets the primary reason people had to overprovision DynamoDB. It changes the entire calculation and obsoletes all the advice you might have heard based on experiences prior to late 2017.
Re: Joy and Pain of Using Google BigTable
#17This matches my experience with BigTable, down to the short-duration failure spikes. I feel that something should be said on the plus side of the ledger here. I'm the solo founder of a company that indexes huge amounts of fine-grained information. Bigtable is the key technology that let me start my company on my own: it soaks up all the data we can throw at it, with almost zero maintenance. Even within the stable of…
Yeah, it does sound like I am really down on the product but it does some things very well. If you can work around the very short lived unavailability it is fantastic.
Re: Joy and Pain of Using Google BigTable
#18We are a user of BigTable, 30k writes/sec and 300k reads/sec, and compared to the other managed services (Pub/Sub, Memorystore, etc), it has been the most stable by far, but we have to scale up our node count at times when we don't think we should have to (based on the perf described in the docs) as well as the latency/errors described in the article. They also added storage caps based on node count last year that in…
Interestingly, for us scaling up doesn't really solve the short term unavailability. It seems to be only somewhat related to load as it does seem to hit more often at high traffic times but we have also seen it at low traffic times.
Putting in that cache is a great move. Cache is challenging for us as we get hits over a very wide range of keys.
Re: Joy and Pain of Using Google BigTable
#19Earlier quoted context omitted.
Yeah, it was quite frustrating trying to figure out what was going on. Up until replication was released, it makes it a real non-starter for a lot of use cases. With replication you can combat the problem and it does give you great performance (when it isn't giving you random errors).
When I was at Google, you were not supposed to serve end users straight out of BigTable. You had to do extra work: request hedging against multiple replicas (Jeff Dean has mentioned this up in public many times, with numbers on long-tail latency impact), some in-memory caching if appropriate, etc. In other words, you had to protect end users from Bigtable: after all, its original target was the web crawling and index…
This is great feedback. Thanks for this. We have heard the same thing particularly about using multiple clusters. They only started offering it last year though!
Re: Joy and Pain of Using Google BigTable
#20Worth noting his original reason for moving away from DynamoDB is outdated. DynamoDB added an “adaptive capacity” feature to handle hot partitions.[1] [1] https://aws.amazon.com/blogs/database/how-amazon-dynamodb-ad...
Yes, I have heard this! Great news they have sorted that out. Might be worth us looking at it again though we are hosted in GCP now so it is less tempting...