Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Inside the longest Atlassian outage

newsletter.pragmaticengineer.com

Post reply on HN
Print view View original

441–450 of 772 posts

Re: Inside the longest Atlassian outage

#441

Post by elxr »

caymanjim wrote: post #384
elesbao wrote: post #376

In a side note that someone else already made: it is interesting to see that many companies that uses JIRA also uses Slack but the noise/complaint/mentions comparing when Slack is down is way different. I barely saw people complaning.

I dunno about everyone else, but I'm generally frustrated and feel blocked when Slack is down, and I celebrate Jira being down because I've never had a pleasant experience using it. Jira is bureaucracy that gets in the way of me getting things done, and Slack is a critical communication path.

Same here. I actively made an effort to tell coworkers how much I hate Jira. Hopefully new startups choose something more sensible.

Re: Inside the longest Atlassian outage

#442

Post by drsim »

beachy wrote: post #411

This is extremely poor for a large SaaS company. A standard RFP question for SaaS should be: - Can you restore data for a single customer, and if so, what is the RTO for that operation? A smaller SaaS could be excused for only thinking about full database restores. When you're a scrappy upstart, thinking about hypotheticals is less important than survival. But for any decent size multi-tenanted SaaS, it's imperative…

Which SaaS platforms provide account-level restores?

If you contact them and say "please restore our data to as it was last week" those I know do not offer this.

Re: Inside the longest Atlassian outage

#443

Post by throwaway894345 »

aaaaaaaaaaab wrote: post #341
systemvoltage wrote: post #78

Earlier quoted context omitted.

What's wrong with C++? Seems unfair to compare it with JIRA.

It's a HackerNews meme from people who never bothered to properly learn C++ and are angry that it's not JavaScript/Ruby/Rust/whatever.

I refer you to my sibling comment: https://news.ycombinator.com/item?id=31017079

Re: Inside the longest Atlassian outage

#444

Post by spookthesunset »

kikki wrote: post #359
jjice wrote: post #336

Earlier quoted context omitted.

Interesting architecture. From a design point of view, I like the idea of full isolation. From an infrastructure point of view I'm a little scared. I'd assume it's actually not that bad and there's a good way to manage the individual DBs and scale them individually. Really interested if you can share any details. Edit: I know each wiki is on a subdomain. Does each wiki also have it's own server?

There are _many_ databases on each server, last I checked there was around 8 servers (or: "clusters") - and we have it so the traffic is somewhat evenly distributed across each server. There are reasonable capacity limits, and when servers get full we spin up a new one and start accepting new wikis there. I am not in OPS, and they do a lot of work behind the scenes to make this all run smoothly - but from an eng pers…

How do you update the schema on 300,000 databases?

Re: Inside the longest Atlassian outage

#445

Post by hotpotamus »

drsim wrote: post #442
beachy wrote: post #411

This is extremely poor for a large SaaS company. A standard RFP question for SaaS should be: - Can you restore data for a single customer, and if so, what is the RTO for that operation? A smaller SaaS could be excused for only thinking about full database restores. When you're a scrappy upstart, thinking about hypotheticals is less important than survival. But for any decent size multi-tenanted SaaS, it's imperative…

Which SaaS platforms provide account-level restores? If you contact them and say "please restore our data to as it was last week" those I know do not offer this.

I actually did this once with Dropbox, though it wasn't a feature they actually published. I clobbered my Dropbox directory accidentally, but I was able to find a script someone wrote to roll it back to a previous point in time and it worked quite well. After that I also took my own snapshots just in case.

Re: Inside the longest Atlassian outage

#446

Post by deckard1 »

hu3 wrote: post #19
pgwhalen wrote: post #14

Earlier quoted context omitted.

> The fact it's been so long and they still haven't revealed and explained the root cause of the outage They did last night: https://www.atlassian.com/engineering/april-2022-outage-upda...

> Faulty script. Second, the script we used provided both the "mark for deletion" capability used in normal day-to-day operations (where recoverability is desirable), and the "permanently delete" capability that is required to permanently remove data when required for compliance reasons. The script was executed with the wrong execution mode and the wrong list of IDs. The result was that sites for approximately 400 cu…

I don't want to assume too much, since the details are sparse. But I know for a fact that few of my current coworkers know a thing about writing tooling code. It's becoming a bit of a lost art.

Here's the way such a script should be done. You have a dry-run flag. Or, better yet, make the script dry-run only. What this script does is it checks the database, gathers actions, and then sends those actions to stdout. You dump this to a file. These commands are executable. They can be SQL, or additional shell scripts (e.g. "delete-recoverable " vs. "delete-permanent ").

The idea is you now have something to verify. You can scan it for errors. You can even put it up on Github for review by stakeholders. You double/triple check the output and then you execute it.

Tooling that enhances visibility by breaking down changes into verifiable commands is incredibly powerful. Making these tools idempotent is also an art form, and important.

Re: Inside the longest Atlassian outage

#447

Post by bzxcvbn »

usefulcat wrote: post #438
Kwpolska wrote: post #428

Earlier quoted context omitted.

> why would you ever delete someone's data when it's literally only 160 KB of text? Compliance? The contract has expired, so there’s no legal basis for them to keep your data?

Seems like that could be addressed with some fine print in the initial agreements. "In the event that you stop paying us, we may keep your data for up to N days unless directed otherwise by you"--or similar.

Why would they bother?

Re: Inside the longest Atlassian outage

#448

Post by robertlagrant »

hnlmorg wrote: post #379
robertlagrant wrote: post #371

Earlier quoted context omitted.

The bug we are mitigating was also just an automation issue.

It’s also pretty easy to foobar up a single DB instance if you don’t have proper guardrails in place. Automation wasn’t the issue here. It’s the symptom not the cause.

No, the symptom was the loss of customer data.

Re: Inside the longest Atlassian outage

#449

Post by spookthesunset »

robertlagrant wrote: post #368
Spivak wrote: post #220

Earlier quoted context omitted.

Having worked at shops that used this architecture it's really not that bad. Can you write the code to do one schema migration? Great, now you can do 1000. App server boots and runs the schema migrations, drops privs and launches the app. Now you've staved off your scaling issues from "how to have a db large enough to hold all our customer data" to "how to have a db large enough to hold our biggest customer's data."…

You can write the code to do 1000 schema migrations, but the problem is if you've migrated 40% of them and hit an issue. What do?

One of the many reasons to put good constrains on fields and use referential integrity! If you don't let the database enforce data validity you are gonna get fucked at some point!

source: every single place I've worked at that poo-poos referential integrity has a database that is full of bullshit that "the application code" never cleaned up

Always use referential integrity. The people who are against it almost always are against it for superstitious reasons (eg: "it makes things slow" or "only one codebase calls it so the code can enforce the integrity"). All it takes is exactly one bug in the application code to corrupt the whole damn thing. And that bug will happen over the lifetime of the product regardless of how "good" or "awesome" the programmers think they are....

... I'll get off my soapbox now!

Re: Inside the longest Atlassian outage

#450

Post by rgj »

heisenbit wrote: post #342
treesknees wrote: post #306

Earlier quoted context omitted.

Good faith would be to lose all of that money to people who are already your customers. Business-wise would be to stay in their good graces and keep those customers by offering the refund, but you don't lose any money to those who either don't care or won't move to a competitor.

Reducing the impact analysis within a long running relationship to a single transaction is too narrow. People observe how other people are treated and draw their conclusions even if not impacted. People may tolerate some abuse but it moves them closer to leaving next time. Money lost in the outage may provide for a budget creation to look for an alternative.

A lot of people making those decisions don’t care about a refund because it’s other people’s money anyway. In my experience only small companies care about that.

Focussing on communicating open and honestly allows them to explain the crap they’re going through because of your mistakes to their bosses, so in fact you can help them save their asses, and they’ll save your ass in return. This is much more important and valuable than a refund.

So you should ALWAYS communicate open and honestly, and offer the refund as an option for clients who do not have a boss to account to.

Post reply on HN