Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

twitter.com

Post reply on HN
Print view View original

141–150 of 260 posts

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#141

Post by andrekandre »

rconti wrote: post #122
breakingcups wrote: post #101

Earlier quoted context omitted.

It's insane. Since they've all but killed off on-premise Jira and Confluence, they've been spamming me regularly trying to convince me to "upgrade" to the cloud. Eventually I gave in and replied to them asking two simple questions. One is that we've had many bad reports from partners that Jira Cloud is incredibly slow, even when compared to the already underperforming Jira Server and I wonder what their performance g…

My employer recently switched from on-prem to cloud. The cloud service is insanely slow, or maybe it's my aging Macbook, but every single component on the page seems to have to load separately. (It's a newer UI versus what we had on-prem). Thankfully we haven't been impacted by this outage.

  > or maybe it's my aging Macbook
on an m1 pro: its slow as molasses

sometimes i have to switch to the "old ui"[0] to get any use out of it (not sure what the cause is, but sometimes its literally unusable)

[0] https://community.atlassian.com/t5/Jira-questions/Re-How-do-...

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#142

Post by bityard »

ghostly_s wrote: post #132
hyperman1 wrote: post #111

Earlier quoted context omitted.

It happened specifically to 1 type of laptop and we only had about 30 of them. So we pulled all of them out of roulation. Then covid struck, so I reformatted most of them with Debian and we gave them away for home schooling. I wonder if I managed to linuxify some kid in the process.

Considering the sorry state of videoconferencing on Linux they probably all immediately had Windows reinstalled.

Not sure what this means, I've been using video conferencing on my Linux laptop for work on a daily basis for the last 5 years.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#143

Post by jeffybefffy519 »

btgeekboy wrote: post #10
lapser wrote: post #4

For those of us not up to date, what exactly has happened? Their status page hasn't actually shown why they're having to rebuild.

Supposedly they’re having to basically restore everyone from backups because a system designed to delete old data was a bit more efficient than it should have been: https://reddit.com/r/sysadmin/comments/u14qqq/_/i4a0mk8/?con...

Or ransomware got them.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#144

Post by alkonaut »

ejb999 wrote: post #140

I use JIRA and confluence every single day, I have to, it is everywhere - but imo it is such a horrific toolset in every way (even before this outage), I can't for the life of me figure out how it got so much market-share.

Because all the others are equally bad, basically. The options for enterprisey issue management are quite slim.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#145

Post by throwaway2048 »

nvr219 wrote: post #85
polski-g wrote: post #74

Earlier quoted context omitted.

Bigger lesson is: do not utilize cloud solutions. Your organization's competency floor is set to the least competent employee of the cloud provider.

I'd disagree with this lesson. Saying "do not utilize cloud solutions" period is nuts. Google and Microsoft are way better at email hosting and delivery than your on-prem server is unless you spend a ton more money on hardware and engineers to keep it up, which is simply not worth it for many companies. Dropbox is going to have better uptime and lower TCO than your self-hosted owncloud instance. What I will say is it…

Good luck being able to use, or even make backups of proprietary SaaS offerings.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#146

Post by bobnamob »

crawl_soever wrote: post #134
senorsmile wrote: post #99

Earlier quoted context omitted.

They are discontinuing on prem after 2022 though. What is your plan if not to migrate to their SaaS offering?

What are disconnected customers going to do? Surely they won't cut off the entire US military and DoD by terminating that cash cow of on premises.

They’re still offering on-prem for airgapped usecases afaik. It’s just become a “contact us” pricing plan

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#147

Post by dijit »

rconti wrote: post #122
breakingcups wrote: post #101

Earlier quoted context omitted.

It's insane. Since they've all but killed off on-premise Jira and Confluence, they've been spamming me regularly trying to convince me to "upgrade" to the cloud. Eventually I gave in and replied to them asking two simple questions. One is that we've had many bad reports from partners that Jira Cloud is incredibly slow, even when compared to the already underperforming Jira Server and I wonder what their performance g…

My employer recently switched from on-prem to cloud. The cloud service is insanely slow, or maybe it's my aging Macbook, but every single component on the page seems to have to load separately. (It's a newer UI versus what we had on-prem). Thankfully we haven't been impacted by this outage.

Can try the native Mac app until it’s discontinued. Surprising how fast it is.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#148

Post by case0x »

breakingcups wrote: post #101
planb wrote: post #68

As the person responsible for running Jira and Confluence on premises at my employer I‘m looking forward for the next time one of their sales droids contacts me to make us move to their cloud services (despite me stating that we are not interested multiple times)…

It's insane. Since they've all but killed off on-premise Jira and Confluence, they've been spamming me regularly trying to convince me to "upgrade" to the cloud. Eventually I gave in and replied to them asking two simple questions. One is that we've had many bad reports from partners that Jira Cloud is incredibly slow, even when compared to the already underperforming Jira Server and I wonder what their performance g…

>Jira Cloud is incredibly slow

I have to say it got better after they switched to AWS but Jira not working/being slow is still an inside joke in the office

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#149

Post by manquer »

mdeeks wrote: post #124
Rantenki wrote: post #46

As someone who is impacted, this is obviously immensely frustrating. Worse, outside of "we have rebuilt functionality for over 35% of the users", I haven't seen any reports from the people who have ostensibly been recovered. Next, their published RTO is 6 hours, so obviously they must have done something that completely demolished their ability to use their standard recovery methods: https://www.atlassian.com/trust/s…

RTO is so hard to state properly even with regular testing. If someone blows away a critical database, sure you can meet your published RTO. What if we lose 300 of our databases and need to copy snapshots from another region. AWS limits you to 20 concurrent snapshot copies cross region. Which of those databases should you do first? Do you know your entire dependency graph for all 1000 of your services to make the rig…

For what you suggest some combination of these things should have happened.

  - Some employee has root access to AWS account and uses it operationally
  - Given wildcard S3 permissions to an IAM user and allowing delete bucket
  - Not enabled object versioning 
  - Cross Region replication not enabled
  - no large bucket protection
  - don't have basic security monitoring and setup of Cloudtail alerts
  - have not invested in full fledged tools for IDS and so on.
If some vendor have any of these issues I don't think any customer would approve these software to be used, these are not normal or best practices .

Large apps have detailed playbooks on how and what gets turned on in what order, and most do DR drills and time those runs periodically. These are well established workflows in any large org.

Yes in a real world downtime you can't have planned for every scenario, maybe you miss the target by 25 % like 2 hours more, or maybe in a very situation you double or even triple it say 12-18 hours. You don't go from 6 to 600+ .

The way RTO is calculated starts by looking at limits on cloud/ hardware / bandwidth/ machine sizes, if basic limits are not factored in like cross region concurrency there is no point in RTO being computed. Even if something like that was missed and you spend tens of millions of dollars on AWS then AWS will work with you and relax those limits .

100x missing the plan either means extremely poor planning or they screwed up something very very badly.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#150

Post by stonogo »

RL_Quine wrote: post #118
jlbooker wrote: post #93

Earlier quoted context omitted.

Restore from off-site tape backup. The kind of service where you ship them ~dozen new tapes in a lockbox each week and they ship you the oldest dozen back. It's supposed to be the "if all of a data centers happen to burn to ashes simultaneously" option. If you say "give us all of our tapes, asap" and then have some pour souls swapping them out as fast as the data can be read... it would probably take a few weeks.

Tapes are not that slow, 360MB/s per drive, and on a large scale swapping them is completely automatic.

In support of your point, 360MB/s is an extremely conservative estimate. I'd expect that from LTO-6, which is around ten years old, and I would certainly hope their backups are on more modern gear than that.
Post reply on HN