Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Inside the longest Atlassian outage

newsletter.pragmaticengineer.com

Post reply on HN
Print view View original

431–440 of 772 posts

Re: Inside the longest Atlassian outage

#431

Post by usefulcat »

notatoad wrote: post #392
a2800276 wrote: post #389

Earlier quoted context omitted.

You can sleep soundly: it seems like they back _everything_ up: > Second, the script we used provided both the "mark for deletion" capability ... (where recoverability is desirable), and the "permanently delete" capability that is required to permanently remove data when required for compliance reasons . The script was executed with the wrong execution mode and the wrong list of IDs. The result was that sites for app…

No, I don't think that's disturbing. That's the point of backups - even when something is permanently and completely erased in the production database, it's still in the backup. Eventually it will get rotated out as the backups expire. Going back and purging things from the backups as part of the delete process would be overdoing it to a ridiculous degree.

> Going back and purging things from the backups as part of the delete process would be overdoing it to a ridiculous degree.

Also, modifying backups is a great way to inadvertently hose your backups.

Re: Inside the longest Atlassian outage

#432

Post by miketria »

Aissen wrote: post #133

The sad truth is that with 99.8% of customers unaffected, it was probably thought to be a minor issue. If those customers didn't have Gergely's ear we probably wouldn't have heard about it.

Hi, this is Mike from Atlassian Engineering. Not a minor issue. Once we knew the extent and severity of the incident, we had hundreds of engineers engaged and working to restore service.

Re: Inside the longest Atlassian outage

#433

Post by a4isms »

shukantpal wrote: post #355
grog454 wrote: post #307

Earlier quoted context omitted.

>"Nobody uses more than 5% of its features, but every company uses a different 5%." >The saying is apocryphal and unlikely to be accurate Well its mathematically impossible to be accurate as soon as you have > 20 users.

False. If you have 100 features, there are nCr(100, 5) combinations of 5% features = 75287520.

If it's no more than 5% of the features, it's actually n-choose-k(100,5) + n-choose-k(100,4) + n-choose-k(100,3) + n-choose-k(100,2) + 100!

    75,287,520
  +  3,921,225 
  +    161,700
  +      4,950
  +        100
  ------------
    79,375,495

Re: Inside the longest Atlassian outage

#434

Post by adamc »

adhesive_wombat wrote: post #400
upbeat_general wrote: post #386

Earlier quoted context omitted.

I don’t believe slack has been down as long? Slack is generally much more critical than JIRA in order to keep working.

It's it though? You can hop onto any of a constellation of other IM platforms, FOSS and not fairly quickly for an instant comms channel, even if you're missing the history. Having all your issue tickets missing is something you can't really deal with unless you have a very recent dump, and even then you can't just fire up Bugzilla and get something working without a lot of migration and administrative effort. You can…

We have all sorts of slack channels set up to coordinate activity, so that internal customers can talk to engineers easily, or engineers can engage with each other. If slack goes down, we'd have to work all that out. For many days, it would be a huge drag on the process, slowing down interactions.

Other IM platforms wouldn't solve that just by existing. Sure, in principle one could set up such channels elsewhere, but that takes time, and the communication about it takes considerably more time.

Re: Inside the longest Atlassian outage

#435

Post by spookthesunset »

throwaway894345 wrote: post #36
h2odragon wrote: post #4

> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production. Simple concepts aren't that simple at their scale, sure, but I suspect this is skimping details on some truly horrendous monolithic architec…

I would really like to understand who makes the decision to purchase JIRA. It's like the C++ of ticketing software--it does everything because no one wanted to sit down and think critically about the use cases and instead decided it would be easier to say "yes" to every single feature request. It definitely feels like whoever is buying JIRA is not on the team who is using it (maybe IT or finance) because it ticks the…

I find it helpful to stop thinking of JIRA as a bug tracker or anything like that. In my opinion JIRA is more of a way to create and track workflows. It can be used as a blank slate for quite a lot of things (which I cannot come up with any examples for at the moment!)

That being said, because it can do anything, it doesn't take much effort to make a workflow as painful as possible. Somebody with the "right" mind might make all kinds of checkpoints in a workflow, which makes a lot of operations a pain in the ass because you wind up hopping through a bunch of steps. Pretty sure in our org we just make our workflow "you can hop from any state to any other state"--basically a free-for-all.

Dunno my point, but there you go!

Re: Inside the longest Atlassian outage

#436

Post by deepspace »

a2800276 wrote: post #389
Alex3917 wrote: post #44

A few years ago we didn't renew our subscription on time because we got the email over Christmas break, and iirc they deleted all of our data in less than two weeks. They were eventually able to manually restore it from backups, but they restored it incorrectly so there was a bunch of stuff broken. This whole thing isn't even remotely surprising to me.

You can sleep soundly: it seems like they back _everything_ up: > Second, the script we used provided both the "mark for deletion" capability ... (where recoverability is desirable), and the "permanently delete" capability that is required to permanently remove data when required for compliance reasons . The script was executed with the wrong execution mode and the wrong list of IDs. The result was that sites for app…

I asked the same question yesterday, and the responses were food for thought.

If you make backups, you are, almost by definition, unable to perform a full 'Compliance Delete' before the oldest backup in the set has expired.

Compliance-based deletion, if it is offered as a service, is almost always something time-based, like "we guarantee the data will be deleted 7 years from now". And then that deliberate deletion step is baked into the backup process.

So, i.m.o. at best they misrepresented the nature of the compliance deletion process. It never did what it was designed to do.

Re: Inside the longest Atlassian outage

#437

Post by gkoberger »

420official wrote: post #257
yabones wrote: post #215

What's a good Jira replacement? Redmine? Phabricator? OpenProject? Just leaving the jira server alone and hoping there's no new and exciting zero-days? One thing is clear, these guys are a bunch of cowboys who can't be trusted with any amount of data.

I'm not at all familiar but a tweet linked from the OP and written by the author plugs https://linear.app/

Linear is phenomenal. Probably built for a different audience than Jira (it's like Superhuman for tickets), but if you want something that works well and is opinionated I highly highly recommend it.

Re: Inside the longest Atlassian outage

#438

Post by usefulcat »

Kwpolska wrote: post #428
Alex3917 wrote: post #141

Earlier quoted context omitted.

Nope. I exported our data after they restored the backup and then we cancelled less than a month later. Like I obviously understand suspending our logins, but why would you ever delete someone's data when it's literally only 160 KB of text? The whole thing made zero sense.

> why would you ever delete someone's data when it's literally only 160 KB of text? Compliance? The contract has expired, so there’s no legal basis for them to keep your data?

Seems like that could be addressed with some fine print in the initial agreements. "In the event that you stop paying us, we may keep your data for up to N days unless directed otherwise by you"--or similar.

Re: Inside the longest Atlassian outage

#439

Post by kitsune_ »

yabones wrote: post #215

What's a good Jira replacement? Redmine? Phabricator? OpenProject? Just leaving the jira server alone and hoping there's no new and exciting zero-days? One thing is clear, these guys are a bunch of cowboys who can't be trusted with any amount of data.

Gitlab would be enough for engineering teams

Re: Inside the longest Atlassian outage

#440

Post by bloopernova »

420official wrote: post #257
yabones wrote: post #215

What's a good Jira replacement? Redmine? Phabricator? OpenProject? Just leaving the jira server alone and hoping there's no new and exciting zero-days? One thing is clear, these guys are a bunch of cowboys who can't be trusted with any amount of data.

I'm not at all familiar but a tweet linked from the OP and written by the author plugs https://linear.app/

Linear has a dark mode. I'm already won over! ;)
Post reply on HN