Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kraken down 24 hours on system upgrade

status.kraken.com

Post reply on HN
Print view View original

51–60 of 61 posts

Re: Kraken down 24 hours on system upgrade

#51

Post by lykr0n »

aneutron wrote: post #48
lykr0n wrote: post #42

Earlier quoted context omitted.

Some upgrades are a one shot deal. If you refactor a data structure, you have to plan to unrefactor it. If you don't, then there are two options, push forward (or hotfix) or restore from backups. Massive schema migrations would be my guess. And I'm willing to bet that their Test/QA Environment only has a small set of test data and not a replication of production

If you run a 700M$ exchange, you should probably have a cold backup of your database every other hour. I think.

At the very least full offline backup before you start.

Re: Kraken down 24 hours on system upgrade

#52

Post by nasalgoat »

whalesalad wrote: post #30

This team is in over their heads. If you don’t have stateless infrastructure with a big red “oops, this isnt right, rollback” button – you’re doing it wrong. Read the reviews of their iPhone app on the App Store. They are not equipped to be running a financial company. Bummer because I’m looking to leave Coinbase. Looks like everyone in this space is nerf-grade.

No kidding - their front page is literally black text on a white background.

They don't have a single person who can use Photoshop and put up a proper maintenance page? They didn't already have a failover maintenance page? This shit is Web Hosting 101 stuff from decades ago.

This is new levels of amateur idiocy.

Re: Kraken down 24 hours on system upgrade

#53

Post by nasalgoat »

RafiqM wrote: post #47

They didn't communicate that there would even be a maintenance to their users. I use Kraken every day and did not receive any sort of email and there wasn't a notice on the site that I could see. Maybe they announced on social media, but that's about it. Starting an unannounced maintenance in the middle of the day is crazy, even if it was only for the original 2 hours. Over 24 hrs now. I don't know if this is due to…

It's clearly incompetence. I suspect someone dropped a table they weren't supposed to and they started scrambling to restore a backup they've never tested and discovered it wasn't working.

Going to kraken.com tells you all you need to know about these folks, all bad.

Re: Kraken down 24 hours on system upgrade

#54

Post by nasalgoat »

eks wrote: post #7
miguelrochefort wrote: post #3

They said it would take 2 hours. It's been more than 24 hours.

It's scary. Why not just revert to the old engine when things started to fell out of place? Having the exchange down for that long is bound to have a big backlash when it comes back online. They should have put a Kraken 2.0 trade engine alongside the first one, and moved people gradually there. It doesn't matter how confident they were with the upgrade before it happened, it's crypto, everything is new. A few lines o…

It boggles my mind that it's in the state it is. Some kid in his basement knows you do staging rollouts and build in parallel, this is seriously the most basic IT knowledge in existence - and yet they still screwed it up, now over 24 hours later.

Whoever their CTO is, is clearly the worst kind of incompetent and the team is the most amateur I have ever seen in my 20 years of Internet systems management.

Re: Kraken down 24 hours on system upgrade

#55

Post by nasalgoat »

eks wrote: post #11
donquichotte wrote: post #9

Earlier quoted context omitted.

It appears that we are at the mercy of mentally unstable amateurs.

Just because the social media guy likes to play social media'ish doesn't mean the rest of the team is like that. But definitely not a good moment to make a joke.

Based on the text-only homepage, I'd say the tech team is much worse - at least the social media guy is amusing.

Re: Kraken down 24 hours on system upgrade

#57

Post by hotsauceror »

This feels like a Resume Generating Event for the DBA team. Sounds like their DR plan doesn't work or doesn't exist. Even if the devs drew up a breaking schema change that wasn't adequately tested and can't be rolled back, the shot-caller would have moved them into disaster recovery mode a long time ago.

Re: Kraken down 24 hours on system upgrade

#58

Post by nasalgoat »

nasalgoat wrote: post #52
whalesalad wrote: post #30

This team is in over their heads. If you don’t have stateless infrastructure with a big red “oops, this isnt right, rollback” button – you’re doing it wrong. Read the reviews of their iPhone app on the App Store. They are not equipped to be running a financial company. Bummer because I’m looking to leave Coinbase. Looks like everyone in this space is nerf-grade.

No kidding - their front page is literally black text on a white background. They don't have a single person who can use Photoshop and put up a proper maintenance page? They didn't already have a failover maintenance page? This shit is Web Hosting 101 stuff from decades ago. This is new levels of amateur idiocy.

I get a downvote for speaking the truth? Someone from the Kraken dev team I assume.

Feel free to explain to me how the current result from http://kraken.com/ is at all even remotely professional.

Re: Kraken down 24 hours on system upgrade

#59

Post by drinchev »

bhouston wrote: post #27
arethuza wrote: post #24

Earlier quoted context omitted.

Saying things like "ahead of the team passing out" in their latest status updates doesn't exactly give confidence that they have either good communications or engineering leadership.

If everyone has been up 24hrs then you will get status updates like that. I guess it is all hands on deck and there isn't a large enough team to do rotations of shifts? Or the issue is very complex and you need all your best guys around to fix it. Yikes. As someone who runs a high availability website, I can feel for them, but it is really bad to be offline this long. I would look to make some changes to that team or…

From what I suspect, we are talking about some backend issue that they are handling. Throughout my career I've never seen a "rotating shifts team" that can handle situations like this. It's usually one guy trying to fix it and a couple of others hanging in the office with a cup of coffee and giving advice on what should the company do next or being forced to update the status page.

Re: Kraken down 24 hours on system upgrade

#60

Post by adamzochowski »

lykr0n wrote: post #42
bhouston wrote: post #18

When upgrades go this bad, can you not just roll back to the previous version? I guess this was some type of breaking change, like a DB re-structure that makes rolling back problematic? And then somehow this wasn't testing enough?

Some upgrades are a one shot deal. If you refactor a data structure, you have to plan to unrefactor it. If you don't, then there are two options, push forward (or hotfix) or restore from backups. Massive schema migrations would be my guess. And I'm willing to bet that their Test/QA Environment only has a small set of test data and not a replication of production

I believe the proper way to upgrade schema is to

1) don't change existing schema (don't alter columns)

2) introduce new schema, side by side (add new columns, all nulls, without defaults)

3) update code to read new schema first, if nothing found, read old schema,

4) create triggers that a write in new or old schema updates the other

5) have a separate process to sequentially migrate data from old schema to new schema

6) once all records are migrated to new schema, start slowly removing code that reads old schema

7) once nothing reads old schema, delete it

Handling schema changes is a tricky process that is spread across multiple different steps, where each step can be rolled back.

Post reply on HN