Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

twitter.com

Post reply on HN
Print view View original

151–160 of 260 posts

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#151

Post by mdoms »

btgeekboy wrote: post #10
lapser wrote: post #4

For those of us not up to date, what exactly has happened? Their status page hasn't actually shown why they're having to rebuild.

Supposedly they’re having to basically restore everyone from backups because a system designed to delete old data was a bit more efficient than it should have been: https://reddit.com/r/sysadmin/comments/u14qqq/_/i4a0mk8/?con...

> This data was from a deprecated service that had been moved into the core datastore of our products.

That is very interesting. This implies they are backing off, at least somewhat, from their very aggressive microservice strategy. Perhaps they feel like they have gone too far in decomposing their products.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#152

Post by joshuaissac »

yabones wrote: post #71
beachy wrote: post #49

Earlier quoted context omitted.

Why would you continue with that plan? You couldn't get a clearer warning signal.

To be quite honest, the problem is historical. We have over a decade of project plans, support tickets, change control logs, etc in our Jira instance. There's simply no painless way to export that into another product that will have approximately the same functionality and features. There's a few that come close, but all fall short of a drop-in replacement. The only options now are the $$$$$ "datacenter" license, mig…

> the $$$$$ "datacenter" license, migrating to the dangerously unstable cloud, or not doing anything and running unsupported EOL software

Data Center is pricier than Server, but isn't it still cheaper than Cloud? And you control your own back-ups as with Server, so Atlassian cannot lose your data.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#153

Post by mdoms »

beart wrote: post #67
bmurphy1976 wrote: post #58

Earlier quoted context omitted.

Bonus tip: don't even type COMMIT until after you've run a few select queries to verify the data has in fact changed how you expected it to change.

super duper bonus tip: Don't execute SQL in a production environment that you just typed out on the fly.

It's incredibly depressing how common this is in the real world, but I can tell you from experience that this NEVER happens in Atlassian production systems.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#154

Post by justin_oaks »

laurent92 wrote: post #47
justin_oaks wrote: post #38

Earlier quoted context omitted.

There are two ways this can go: 1) This outage will get their organization to prioritize work such that it never happens again. 2) This outage is representative of a dysfunctional organization that can't prioritize work correctly. If you've been using Atlassian software for a while and are used to how they prioritize tickets then one of those options seems far more likely than the other.

> 1) This outage will get their organization to prioritize work such that it never happens again It has already happened in the past.

Ah, you mean that outages have happened in the past, not that Atlassian has changed because of an outage.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#155

Post by raffraffraff »

m0llusk wrote: post #112
perlgeek wrote: post #66

So, let me get this straight: * It's been deleted for a week already, they estimate they might need two more weeks. Three in total. * They claim to have "extensive backups", and hundreds of engineers working on it. What? How? This simply doesn't go together. Why would restoring from backup take three weeks? Either their backups aren't complete, or they need new software written for the restore, or something else does…

If you are not regularly testing restores then you don't have backups.

I'd say this is the answer.

I remember a situation where we had a near miss with data loss (replica failed and master had a bad disk). We didn't want to put the production database under extra load by taking a live backup while it was handling all production traffic, so we restored a backup. But it was "bad". Tried the one before it, and the one before that. Apparently they were busted for over a month due to a config change. We restored a month-old backup and started applying binlogs (which thankfully we had been backing up). But that meant replaying a month of transactions into the restored database. I can't remember the details but I think we ended up replacing the bad disk, resilvering the array and live-cloning the primary before the binlogs got fully applied to the one we restored from the old backup.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#156

Post by nvr219 »

josephcsible wrote: post #125
nvr219 wrote: post #85

Earlier quoted context omitted.

I'd disagree with this lesson. Saying "do not utilize cloud solutions" period is nuts. Google and Microsoft are way better at email hosting and delivery than your on-prem server is unless you spend a ton more money on hardware and engineers to keep it up, which is simply not worth it for many companies. Dropbox is going to have better uptime and lower TCO than your self-hosted owncloud instance. What I will say is it…

There's something humorous when better uptime is cited as a reason to use cloud software instead of on-prem in a thread about a major cloud outage.

I mean, yes, your point is well taken, but I stand by my point nonetheless.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#157

Post by nvr219 »

throwaway2048 wrote: post #145
nvr219 wrote: post #85

Earlier quoted context omitted.

I'd disagree with this lesson. Saying "do not utilize cloud solutions" period is nuts. Google and Microsoft are way better at email hosting and delivery than your on-prem server is unless you spend a ton more money on hardware and engineers to keep it up, which is simply not worth it for many companies. Dropbox is going to have better uptime and lower TCO than your self-hosted owncloud instance. What I will say is it…

Good luck being able to use, or even make backups of proprietary SaaS offerings.

You can make backups of the data absolutely. There's a whole industry selling cloud-to-cloud backup solutions.

Some examples: https://www.ownbackup.com/ https://www.backupify.com/ https://www.cloudally.com/

Edit: to clarify I would say if the data is important to you, then “the ability to back up the data” should be a requirement when selecting saas. See my other comment in this thread on ms planner.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#158

Post by Trasmatta »

justin_oaks wrote: post #64
xenadu02 wrote: post #29

Earlier quoted context omitted.

Reminder: never delete data for real as your first step. Always mark it deleted along with a time stamp saying when. Then you can hide deleted itemsfrom everything. When a maintenance script goes haywire you can fix the problem quickly. Have a daily job that really deletes records marked deleted after 30 days. If that is too complicated to retrofit then have any mass cleanup script move the records to a CSV file or t…

I agree that data must never be "deleted and forever gone" unless you've already been very sure about it a few time. But I would like to warn people about certain implementations of database "soft deletes" that I'm not a fan of. To be clear, I'm talking about the idea of having a "deleted" and/or a "date_deleted" column and using those columns in the WHERE clause to filter out rows that shouldn't be visible. That pat…

Agreed, soft deleting adds so much complexity to everything. And even has the potential for privacy related bugs. Like, say, accidentally forgetting to respect the deleted column in a query on a joining table that determines user permissions for some resource. Now people have access to something they had permissions revoked for. Whoops.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#159

Post by manquer »

nvr219 wrote: post #85
polski-g wrote: post #74

Earlier quoted context omitted.

Bigger lesson is: do not utilize cloud solutions. Your organization's competency floor is set to the least competent employee of the cloud provider.

I'd disagree with this lesson. Saying "do not utilize cloud solutions" period is nuts. Google and Microsoft are way better at email hosting and delivery than your on-prem server is unless you spend a ton more money on hardware and engineers to keep it up, which is simply not worth it for many companies. Dropbox is going to have better uptime and lower TCO than your self-hosted owncloud instance. What I will say is it…

Have your own backup works only if there are common open standards for export/ import. Email or storage may have those, project management tools don't. I can't simply backup from Jira and start using pivotal.

Even for Email or storage or any other open system, UX changes and feature differences can take a lot of time to train properly, you don't migrate from one vendor to another vendor just like that.

Re: Atlassian: We estimate the rebuilding effort to last for up to 2 more weeks

#160

Post by Trasmatta »

danuker wrote: post #43
xenadu02 wrote: post #29

Earlier quoted context omitted.

Reminder: never delete data for real as your first step. Always mark it deleted along with a time stamp saying when. Then you can hide deleted itemsfrom everything. When a maintenance script goes haywire you can fix the problem quickly. Have a daily job that really deletes records marked deleted after 30 days. If that is too complicated to retrofit then have any mass cleanup script move the records to a CSV file or t…

> or bad SQL WHERE clause Tip: Begin an SQL session with BEGIN TRANSACTION; at the end you can either COMMIT or ROLLBACK.

Another tip: never enter queries directly in a production database connection with write access in the first place. (Ideally very few people even have that level of access.) Write it in your codebase, write tests for it, get it code reviewed, and run it in a dry run first and get a list of affected records before running it for real.
Post reply on HN