TLDR: MongoDB
There’s definitely some truth to parent’s snark:
Coinbase Incident Post Mortem: June 25–26, 2019
21–30 of 30 posts
Re: Coinbase Incident Post Mortem: June 25–26, 2019
#22> We take uptime very seriously, and we’re working hard to support the millions of customers that choose Coinbase to manage their cryptocurrency No you don't. - If you did, you'd hire a DBA team and they would be familiar with the various jobs in your environment. But first your founders would have to have respect for Operations, which will take a dozen more major outages. The other major Coinbase outages have also b…
I think this is exactly correct, and I hate that this behavior of offloading testing onto users seems more and more to just be the expectation rather than the exception.
Coinbase had a number of issues when cryptocurrencies really exploded in 2017, and at that time I felt more willing to give them the benefit of the doubt because the landscape of cryptocurrencies had shifted so dramatically and I could empathize with their struggles to keep up. Two years later, there aren't any excuses anymore in my mind. As the parent comment says -- it's just not a priority.
I've happily moved off the platform to other options which have given me no trouble whatsoever.
Re: Coinbase Incident Post Mortem: June 25–26, 2019
#23Earlier quoted context omitted.
>you wouldn't be doing major database (or other production) changes at 3 pm in the afternoon. Coinbase isn't dedicated to one timezone. They fully support tons of countries and have customers all over the place. Things like this have to happen at some point, and there are benefits to doing stuff like this during "work" hours (like having all of your staff online and available)
> Things like this have to happen at some point, and there are benefits to doing stuff like this during "work" hours (like having all of your staff online and available) That sounds like a benefit to Coinbase and not to any of their customers.
Eh, if Coinbase were the stock market and only open for trading for < 8 hours a day, I'd ding them, but it's always 3pm somewhere. 3pm PDT is 6pm EDT is 10pm UTC.
Re: Coinbase Incident Post Mortem: June 25–26, 2019
#24Earlier quoted context omitted.
> Things like this have to happen at some point, and there are benefits to doing stuff like this during "work" hours (like having all of your staff online and available) That sounds like a benefit to Coinbase and not to any of their customers.
Eh, if Coinbase were the stock market and only open for trading for < 8 hours a day, I'd ding them, but it's always 3pm somewhere. 3pm PDT is 6pm EDT is 10pm UTC.
One can only hope that they know when their peak trading patterns generally are.
Re: Coinbase Incident Post Mortem: June 25–26, 2019
#25Earlier quoted context omitted.
>you wouldn't be doing major database (or other production) changes at 3 pm in the afternoon. Coinbase isn't dedicated to one timezone. They fully support tons of countries and have customers all over the place. Things like this have to happen at some point, and there are benefits to doing stuff like this during "work" hours (like having all of your staff online and available)
> Things like this have to happen at some point, and there are benefits to doing stuff like this during "work" hours (like having all of your staff online and available) That sounds like a benefit to Coinbase and not to any of their customers.
The inverse (not being able to do this during work hours) means that if you find an issue you won't be able to roll out a fix in an expedited manner.
Being able to fix things while they are running is a big advantage, and tends to make the entire stack lean more towards something resilient to partial failure
Re: Coinbase Incident Post Mortem: June 25–26, 2019
#26> We take uptime very seriously, and we’re working hard to support the millions of customers that choose Coinbase to manage their cryptocurrency No you don't. - If you did, you'd hire a DBA team and they would be familiar with the various jobs in your environment. But first your founders would have to have respect for Operations, which will take a dozen more major outages. The other major Coinbase outages have also b…
>you wouldn't be doing major database (or other production) changes at 3 pm in the afternoon. Coinbase isn't dedicated to one timezone. They fully support tons of countries and have customers all over the place. Things like this have to happen at some point, and there are benefits to doing stuff like this during "work" hours (like having all of your staff online and available)
> Coinbase isn't dedicated to one timezone. They fully support tons of countries and have customers all over the place.
What timezone are their developers in? That’s the important question in this situation.
Devs deploying changes that affect customers at the end of the devs’ workday are reckless.
Re: Coinbase Incident Post Mortem: June 25–26, 2019
#27I suppose this is a good time to ask whether Coinbase thinks they are a bank, or a brokerage, or both.
If they are a bank, this isn't the end of the world. I've had online banking outages at "normal" banks. It is still a bad thing, but there are other ways I can get my money, like going to a branch.
On the other hand, if Coinbase is like a brokerage, this is really bad. And let's face it, most use of crypto is for investment and speculation purposes. For trades to fail for half an hour is really bad. If they are running this thing like a startup on MongoDB (seriously?) I don't see how anyone who puts their money in can have any confidence of getting it back out.
Re: Coinbase Incident Post Mortem: June 25–26, 2019
#28I suppose this is a good time to ask whether Coinbase thinks they are a bank, or a brokerage, or both. If they are a bank, this isn't the end of the world. I've had online banking outages at "normal" banks. It is still a bad thing, but there are other ways I can get my money, like going to a branch. On the other hand, if Coinbase is like a brokerage, this is really bad. And let's face it, most use of crypto is for in…
> If they are running this thing like a startup on MongoDB (seriously?)
Do you base this on recent info (latest versions w/ Jepsen tests)? If so, what specifically makes Mongo a "startup" db?
Re: Coinbase Incident Post Mortem: June 25–26, 2019
#29Re: Coinbase Incident Post Mortem: June 25–26, 2019
#30Earlier quoted context omitted.
There’s definitely some truth to parent’s snark: http://hackingdistributed.com/2013/01/29/mongo-ft/
Really, rolling out the Mongo spite from '13? Isn't there a statute of limitations on this sort of pgres fanboy balderdash? Haters gonna hate, hate, hate
Even SQLite doesn’t lose data.
This “spite” reveals serious core design flaws in Mongo that still have yet to be addressed.
If someone has data showing otherwise — that the abject reliability and scaling issues were resolved via something like a complete rewrite, I’d be glad to reconsider my views (they won’t have any though).