Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

twitter.com

Post reply on HN
Print view View original

221–230 of 260 posts

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#221

Post by manquer »

bigiain wrote: post #195
manquer wrote: post #149

Earlier quoted context omitted.

For what you suggest some combination of these things should have happened. - Some employee has root access to AWS account and uses it operationally - Given wildcard S3 permissions to an IAM user and allowing delete bucket - Not enabled object versioning - Cross Region replication not enabled - no large bucket protection - don't have basic security monitoring and setup of Cloudtail alerts - have not invested in full…

> 100x missing the plan either means extremely poor planning or they screwed up something very very badly. Like most airliner accidents, this is probably an unfortunate combination of both of those things happening at the same time. My guess would be they have fairly decent planning overall but there's one (or more) small-ish areas where their planning is extremely poor - which crossed over with a screwup in a very s…

Aviation has paid with so much blood resulting in tight regulations and internal controls to get the point where any current accidents are unlikely edge case scenarios. [1]

SaaS has lot more tolerance for failure, so my money is it is something simpler but difficult to get implemented in large org.

---

In a ideal world this incident should impact their revenue, growth and stock price substantially.

It is unlikely to do so, because of stickiness of enterprise customers, no better alternatives, compared to say Google, Facebook, Amazon where a minute of downtime is immediate quantifiable revenue loss so FAANG really obsess so much over how many 9s of uptime.

The typical management of enteripse app companies like Atlassian have no incentive to do anything beyond cursory lip service and get away with under investing in tech.

---

[1]3 years back i would have stood by that, but after Boeing 737 max twin disaster and systematic problems leading to it , i am not so sure those lessons are not forgotten.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#222

Post by abrookewood »

Meanwhile, people have been waiting since at least 2013 for Atlassian to deliver a way to automate backups for their Cloud offerings: https://jira.atlassian.com/browse/CLOUD-6498

And there's still very little movement: https://community.atlassian.com/t5/Backup-Restore-articles/E...

But don't worry! It's in the Cloud! It's all fine!

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#223

Post by [deleted] »

Nextgrid wrote: post #83
perlgeek wrote: post #66

So, let me get this straight: * It's been deleted for a week already, they estimate they might need two more weeks. Three in total. * They claim to have "extensive backups", and hundreds of engineers working on it. What? How? This simply doesn't go together. Why would restoring from backup take three weeks? Either their backups aren't complete, or they need new software written for the restore, or something else does…

My theory in my other comment is that they've deleted some data and are waiting on third-party data recovery specialists. That would explain the timescale.

[deleted]

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#224

Post by KennyBlanken »

RL_Quine wrote: post #118
jlbooker wrote: post #93

Earlier quoted context omitted.

Restore from off-site tape backup. The kind of service where you ship them ~dozen new tapes in a lockbox each week and they ship you the oldest dozen back. It's supposed to be the "if all of a data centers happen to burn to ashes simultaneously" option. If you say "give us all of our tapes, asap" and then have some pour souls swapping them out as fast as the data can be read... it would probably take a few weeks.

Tapes are not that slow, 360MB/s per drive, and on a large scale swapping them is completely automatic.

And that's just one drive. I ran a piddling departmental backup server and we had a dual-drive library.

Four drive libraries are pretty common, too.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#225

Post by x0x0 »

Nextgrid wrote: post #26

I bet they screwed up royally, deleted some data and are down to either rebuilding it from logs, caches or other side-effects, or using data recovery software on the storage drives (which might involve third-party companies). I can't see many other reasons why this should take 2 weeks.

Everyone who's used jira understands that they don't hire good engineers. If they did, it wouldn't be So. Damn. Slow.

They're almost certainly rebuilding something from scratch.

If it were an AWS system limitation, almost all of those can be lifted if you ask nicely and are a big account.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#226

Post by cryptonector »

perlgeek wrote: post #66

So, let me get this straight: * It's been deleted for a week already, they estimate they might need two more weeks. Three in total. * They claim to have "extensive backups", and hundreds of engineers working on it. What? How? This simply doesn't go together. Why would restoring from backup take three weeks? Either their backups aren't complete, or they need new software written for the restore, or something else does…

Restoring everything from backups is very hard.

Something that can make restore-from-backups harder, and that I've seen happen, is when the backup/restore systems themselves get destroyed by the same black swan event. Then you have to first recover those by doing fresh installs, and you have to have all the people on hand who know what the configurations would have been to be able to then use the backup library. Then you have to begin restoring a few target systems to check that everything is OK with the restore process, then you have to restore everything though you'll be limited by the restore system's bandwidth.

How could this happen? Well, a disgruntled employee could make it happen. It happened at Paine Webber in 2002 [0]. In that case the attacker left a time bomb in the boot process on all systems they could reach, and that included the backup/restore servers. Worse, the time bomb was in the backups themselves, so restored systems ate themselves as soon as they were booted, which slowed down the recovery process.

  [0] https://www.independent.co.uk/news/business/news/disgruntled-worker-tried-to-cripple-ubs-in-protest-over-32-000-bonus-481515.html
  
      https://www.justice.gov/archive/criminal/cybercrime/press-releases/2002/duronioIndict.htm

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#227

Post by Too »

justin_oaks wrote: post #64
xenadu02 wrote: post #29

Earlier quoted context omitted.

Reminder: never delete data for real as your first step. Always mark it deleted along with a time stamp saying when. Then you can hide deleted itemsfrom everything. When a maintenance script goes haywire you can fix the problem quickly. Have a daily job that really deletes records marked deleted after 30 days. If that is too complicated to retrofit then have any mass cleanup script move the records to a CSV file or t…

I agree that data must never be "deleted and forever gone" unless you've already been very sure about it a few time. But I would like to warn people about certain implementations of database "soft deletes" that I'm not a fan of. To be clear, I'm talking about the idea of having a "deleted" and/or a "date_deleted" column and using those columns in the WHERE clause to filter out rows that shouldn't be visible. That pat…

I guess one could make a view for each table that always includes the where deleted = false, to not bother about it in application code. Still yes, it adds complexity.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#228

Post by Too »

perlgeek wrote: post #66

So, let me get this straight: * It's been deleted for a week already, they estimate they might need two more weeks. Three in total. * They claim to have "extensive backups", and hundreds of engineers working on it. What? How? This simply doesn't go together. Why would restoring from backup take three weeks? Either their backups aren't complete, or they need new software written for the restore, or something else does…

My guess is they failed halfway through a major schema or api migration. If some of the services have already progressed too far, then rolling back another service to previous backup snapshot will make the two incompatible. Especially if one of the services is global and the other is per customer.

The only way out is to figure out the bugs and continue migrating forward, fixing issues as they appear one by one.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#229

Post by jamesfinlayson »

ejb999 wrote: post #140

I use JIRA and confluence every single day, I have to, it is everywhere - but imo it is such a horrific toolset in every way (even before this outage), I can't for the life of me figure out how it got so much market-share.

It launched in 2002 - well before my time but I'm guessing that for the first few years of its life there weren't many competitors.

Now it has endless competitors and I'm led to believe that it has accumulated lots of features that businesses can't live without but which the average end user never touches.

Post reply on HN