2-weeks is one hell of a Recovery-Time-Objective.
Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks
21–30 of 260 posts
Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks
#22We're mere weeks aware from migrating to their could platform after the self-hosted rugpull. This really doesn't give me confidence in their ability to not break my stuff.
Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks
#23It's funny that they recently killed self-hosted Jira. If you'd self-hosted you'd be fine.
We're using the cloud version and we're fine too (no outage). What's your point? Are you claiming that self-hosted is never down? Or that self-hosted is more reliable? Because I doubt that. Difference is just that when self-hosted goes down, it doesn't end up in the news.
It sounds like Atlassian is doing individual restores, each restore takes a fair amount of time, and they don't have the capacity to do all 400 simultaneously (because why would they). So you just have to wait.
If you're self-hosted, you dedicate as many people as possible/necessary to restoring your service, and it becomes their top priority.
You also have a lot more insight into the detailed inner workings of the restore, making it easier to plan against, instead of just vague "we're working on it" messages for days a time with no clear end in sight.
Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks
#24I'm honestly surprised their stock hasn't plummeted yet. There is no way I could ever invest in a company that was this terrible.
If technical competency had any bearing on stock prices they should've been at 0 since long ago. Their stock price is tied to the amount of clueless/shitty companies that will still cling onto their products regardless of what happens, and I don't think this incident is going to change much.
Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks
#25It's funny that they recently killed self-hosted Jira. If you'd self-hosted you'd be fine.
To be fair, Jira screams SaaS. I don't want any of my company trapped on it, but if they were I'm sure as well not going to self host that spawn of hell.
Self-hosted full Atlassian stack (jira, confluence, bamboo, bitbucket) for 6+ years. ~200k tickets. To be honest, it just runs with minimal issues.
Mostly downtime is just upgrades. I can remember a few times we've had to add (JVM) memory as our usage increased. Not sure what we're going to do with the discontinuation of server product line. We self-host to keep source code, etc. more than one configuration mistake (or zero-day) away from exposing it to the world.
Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks
#26I bet they screwed up royally, deleted some data and are down to either rebuilding it from logs, caches or other side-effects, or using data recovery software on the storage drives (which might involve third-party companies). I can't see many other reasons why this should take 2 weeks.
Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks
#27It's funny that they recently killed self-hosted Jira. If you'd self-hosted you'd be fine.
We're using the cloud version and we're fine too (no outage). What's your point? Are you claiming that self-hosted is never down? Or that self-hosted is more reliable? Because I doubt that. Difference is just that when self-hosted goes down, it doesn't end up in the news.
My own self-hosted services have about 20 min downtime a month.
I’ve never – not at any point in the past 10 years – gone over 24h of downtime.
JIRA will now have 3 weeks downtime.
Distributed systems have complexity that grows superlinear, which leads to more and longer incidents.
Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks
#28Earlier quoted context omitted.
I think there is a strange psychological trait that I have, and others may as well, where I am much more forgiving breaking my own stuff than having someone else do it.
People are more forgiving torwards themselves and their own folks, I can understand that. I'm just thinking it's important to make decisions based on facts. Some self-hosters walk around with this "my own basement is safer than Amazon datacenters" attitude and that's just not true (in most cases, I guess :D).
In my years of selfhosting personal stuff I never had a service “sunset“, never lost data, and never had significant downtime.
In the external services I use, downtime of one service or other is to be expected at least a few times a year, and the “sunsets” happen occasionally.
Thing is, public services are solving a much more difficult problem (keeping things running safely for millions).
Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks
#29For those of us not up to date, what exactly has happened? Their status page hasn't actually shown why they're having to rebuild.
Supposedly they’re having to basically restore everyone from backups because a system designed to delete old data was a bit more efficient than it should have been: https://reddit.com/r/sysadmin/comments/u14qqq/_/i4a0mk8/?con...
Reminder: never delete data for real as your first step. Always mark it deleted along with a time stamp saying when. Then you can hide deleted itemsfrom everything. When a maintenance script goes haywire you can fix the problem quickly. Have a daily job that really deletes records marked deleted after 30 days.
If that is too complicated to retrofit then have any mass cleanup script move the records to a CSV file or temporary table.
Never ever ever be in a situation where a rogue script or bad SQL WHERE clause means restoring from backups.
Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks
#30[deleted]