Kraken down 24 hours on system upgrade • Hacker News Board

Re: Kraken down 24 hours on system upgrade

#41

Post by liquidise » Fri, Jan 12, 2018, 1:12 PM UTC

rando444 wrote: post #32

edf13 wrote: post #25

Never, ever, not ever... Run major updates to production... Near a weekend. Surely they're not going to complete the go-live later today are they?? I'm assuming of course all of this as been heavily tested on test networks for a couple weeks... and of course this is just some deployment issue... (Sarcasm)

Speaking in general terms, not running updates near a weekend is only so you know you have personnel on hand to fix problems. If you have the people available, then weekends are best, because it's usually a lower impact for end users.

This varies industry by industry. As someone who runs a dating site, we've all but outlawed non-trivial Friday releases because Fri-Sun are peak days.

It wouldn't surprise me if crypto volumes saw a similar trend... though in reality i suspect people account for a minority of volume when compared to bots.

Re: Kraken down 24 hours on system upgrade

#42

Post by lykr0n » Fri, Jan 12, 2018, 1:15 PM UTC

bhouston wrote: post #18

When upgrades go this bad, can you not just roll back to the previous version? I guess this was some type of breaking change, like a DB re-structure that makes rolling back problematic? And then somehow this wasn't testing enough?

Some upgrades are a one shot deal. If you refactor a data structure, you have to plan to unrefactor it. If you don't, then there are two options, push forward (or hotfix) or restore from backups.

Massive schema migrations would be my guess. And I'm willing to bet that their Test/QA Environment only has a small set of test data and not a replication of production

Re: Kraken down 24 hours on system upgrade

#43

Post by rburhum » Fri, Jan 12, 2018, 1:15 PM UTC

eks wrote: post #11

donquichotte wrote: post #9

Earlier quoted context omitted.

It appears that we are at the mercy of mentally unstable amateurs.

Just because the social media guy likes to play social media'ish doesn't mean the rest of the team is like that. But definitely not a good moment to make a joke.

The amount of downtime and how their wire has been for months says that

Re: Kraken down 24 hours on system upgrade

#44

Post by JorgeGT » Fri, Jan 12, 2018, 1:16 PM UTC

dr_zoidberg wrote: post #39

JorgeGT wrote: post #6

They erased this glorious status update: > I fell asleep at my desk. In a dream, the ghost of a butterfly from the future whispered to me "the dollar hit 0 bitcoin". I wake up, people around me dabbing the blood from their eyes. Progress has been made but not enough. I crack my whip -- "we aren't getting to zero like this!". The flogging will continue until morale improves. Expect another update after I make my round…

Guess in his dream BTC was over 33.5M and the butterfly was only using float16 precision...

Besides their tendency to create antipodal hurricanes, type safety is a major problem with butterflies.

Re: Kraken down 24 hours on system upgrade

#45

Post by forthefuture » Fri, Jan 12, 2018, 1:20 PM UTC

dx034 wrote: post #29

miguelrochefort wrote: post #26

Earlier quoted context omitted.

Billions.

Billions for both or just volume? Money in accounts would be interesting as that's where regulators (or the FBI) will step in at some point if it looks like potential fraud.

No one releases money in accounts numbers.

Kraken was doing $711M in daily volume before the maintenance. [1]

Only the top 8 exchanges have cracked a billion in daily volume. [2]

[1] https://coinmarketcap.com/exchanges/kraken/

[2] https://coinmarketcap.com/exchanges/volume/24-hour/all/

Re: Kraken down 24 hours on system upgrade

#46

Post by wonderwonder » Fri, Jan 12, 2018, 1:29 PM UTC

dx034 wrote: post #28

stp-ip wrote: post #15

Earlier quoted context omitted.

Second the notion of a post mortem, but I assume due to their focus on security they value security by obscurity as an additional factor.

Explaining why an upgrade didn't work doesn't compromise security. I'd guess moving the data didn't work or they corrupted a database and don't know how to repair it.. Explaining that you don't have proper data backups in place can be embarrassing but with a post mortem you can at least get some trust back (like Gitlab's incident). Not explaining why you're offline for 24h doesn't help people to trust you

If they corrupted a live database and are not able to recover it they are in a world of hurt. While it is bad form, many people keep their coins on the exchanges and even if the bulk of an individuals coins are offline, they still likely have at least a small amount on their for trading.

If a table that connects user accounts to kraken owned wallets is corrupted and not recoverable people will be out millions. For some that would be the equivalent of your 401k issuing a post mortem for losing all of your retirement.

If this worst case scenario happened they are likely in severe damage control.

Most likely explanation though is that things are just taking longer than expected to upgrade what is by all measures likely a very technical and convoluted system.

Re: Kraken down 24 hours on system upgrade

#47

Post by RafiqM » Fri, Jan 12, 2018, 1:40 PM UTC

They didn't communicate that there would even be a maintenance to their users. I use Kraken every day and did not receive any sort of email and there wasn't a notice on the site that I could see. Maybe they announced on social media, but that's about it.

Starting an unannounced maintenance in the middle of the day is crazy, even if it was only for the original 2 hours. Over 24 hrs now.

I don't know if this is due to incompetence or intentional.

If they announced the site was going down for major upgrade the smart thing for users to do would be to transfer their crypto to their offline wallets. Possibly withdraw their fiat also. This is what many do for a major fork of a single coin.

Separately, there is for sure going to be tons of money lost because Kraken supports margin trading and the market took a 15% dive right around this time, I suspect there are at least a few that got a margin call and are screwed. These people would have been able to manage their positions if the service didn't have extended and unannounced downtime.

Re: Kraken down 24 hours on system upgrade

#48

Post by aneutron » Fri, Jan 12, 2018, 2:12 PM UTC

lykr0n wrote: post #42

bhouston wrote: post #18

When upgrades go this bad, can you not just roll back to the previous version? I guess this was some type of breaking change, like a DB re-structure that makes rolling back problematic? And then somehow this wasn't testing enough?

Some upgrades are a one shot deal. If you refactor a data structure, you have to plan to unrefactor it. If you don't, then there are two options, push forward (or hotfix) or restore from backups. Massive schema migrations would be my guess. And I'm willing to bet that their Test/QA Environment only has a small set of test data and not a replication of production

If you run a 700M$ exchange, you should probably have a cold backup of your database every other hour. I think.

Re: Kraken down 24 hours on system upgrade

#49

Post by crypticjf » Fri, Jan 12, 2018, 2:37 PM UTC

As a long time software developer and manager, it is unimaginable to me that they had not done proper analysis of the risks and had mitigation strategies in place. The whole future butterfly comment was bizarre and I saw it when it happened. I am hoping at the very least they have DB backups but I would think if they did, they would already have been restored. With each passing hour I am in fact growing very concerned. Not to mention the run on that place could likely put them out of business when and if they ever do come back online. Just look at the owner over there. Smoking too much weed I suspect?

Re: Kraken down 24 hours on system upgrade

#50

Post by libx » Fri, Jan 12, 2018, 2:53 PM UTC

I had an account with them some months ago, but their system is like shit. Completely shit. When I was long, the balance showed the opposite, when the market was against me my position improved and vice-versa. This had all the bad consequences that one can imagine, as I couldn't close some positions because supposedly there were no funds for that, when in fact, there were.

They need a system fix, I really don't trust them.