Live data from Hacker News

Kraken down 24 hours on system upgrade

status.kraken.com

21–30 of 61 posts

Re: Kraken down 24 hours on system upgrade

#21

The graphs at the bottom seem to be telling a different story?

A website with no functionality responds quickly and the API graph can be up 100% if it only returns a static message (didn't test their API). But they should've removed that API chart, a bit misleading.

Re: Kraken down 24 hours on system upgrade

#22
post #6

They erased this glorious status update: > I fell asleep at my desk. In a dream, the ghost of a butterfly from the future whispered to me "the dollar hit 0 bitcoin". I wake up, people around me dabbing the blood from their eyes. Progress has been made but not enough. I crack my whip -- "we aren't getting to zero like this!". The flogging will continue until morale improves. Expect another update after I make my round…

It appears that we are at the mercy of mentally unstable amateurs.

[deleted]

Re: Kraken down 24 hours on system upgrade

#24
post #11

Earlier quoted context omitted.

It appears that we are at the mercy of mentally unstable amateurs.

Just because the social media guy likes to play social media'ish doesn't mean the rest of the team is like that. But definitely not a good moment to make a joke.

Saying things like "ahead of the team passing out" in their latest status updates doesn't exactly give confidence that they have either good communications or engineering leadership.

Re: Kraken down 24 hours on system upgrade

#25
Never, ever, not ever...

Run major updates to production...

Near a weekend. Surely they're not going to complete the go-live later today are they??

I'm assuming of course all of this as been heavily tested on test networks for a couple weeks... and of course this is just some deployment issue... (Sarcasm)

Re: Kraken down 24 hours on system upgrade

#27
post #11

Earlier quoted context omitted.

Just because the social media guy likes to play social media'ish doesn't mean the rest of the team is like that. But definitely not a good moment to make a joke.

Saying things like "ahead of the team passing out" in their latest status updates doesn't exactly give confidence that they have either good communications or engineering leadership.

If everyone has been up 24hrs then you will get status updates like that.

I guess it is all hands on deck and there isn't a large enough team to do rotations of shifts? Or the issue is very complex and you need all your best guys around to fix it.

Yikes.

As someone who runs a high availability website, I can feel for them, but it is really bad to be offline this long. I would look to make some changes to that team or address an unstaffing issue or something. This should not happen. This should have been preventable/foreseeable and a risk mitigation strategy could have been in place.

Re: Kraken down 24 hours on system upgrade

#28
post #15

Earlier quoted context omitted.

I notice the "upgrade is coming" notice on their site last week, but I don't have a Kraken account, I was just interested in learning more about crypto-currency trading. Now they made me curious about their platform, for a purely technical perspective. As you say, why didn't they just revert? Are they not able to? What are the steps in their system upgrade? Are they moving to new hardware? What's their setup like? Wh…

Second the notion of a post mortem, but I assume due to their focus on security they value security by obscurity as an additional factor.

Explaining why an upgrade didn't work doesn't compromise security. I'd guess moving the data didn't work or they corrupted a database and don't know how to repair it.. Explaining that you don't have proper data backups in place can be embarrassing but with a post mortem you can at least get some trust back (like Gitlab's incident).

Not explaining why you're offline for 24h doesn't help people to trust you

Re: Kraken down 24 hours on system upgrade

#29

Earlier quoted context omitted.

How much do they have under management or their daily volume, etc?

Billions.

Billions for both or just volume? Money in accounts would be interesting as that's where regulators (or the FBI) will step in at some point if it looks like potential fraud.

Re: Kraken down 24 hours on system upgrade

#30
This team is in over their heads. If you don’t have stateless infrastructure with a big red “oops, this isnt right, rollback” button – you’re doing it wrong. Read the reviews of their iPhone app on the App Store. They are not equipped to be running a financial company.

Bummer because I’m looking to leave Coinbase. Looks like everyone in this space is nerf-grade.

Post reply on HN