Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Inside the longest Atlassian outage

newsletter.pragmaticengineer.com

Post reply on HN
Print view View original

531–540 of 772 posts

Re: Inside the longest Atlassian outage

#531

Post by fphhotchips »

taude wrote: post #509
scottlamb wrote: post #504

Earlier quoted context omitted.

Is it standard for a RFP to have a long list of questions like this? I've never been involved in an RFP from either side. Is it standard to (in addition or instead) to have something more general/forward-looking like: how do you watch other providers' postmortems and apply the lessons to your own system? > - Can you restore data for a single customer, and if so, what is the RTO for that operation? If I were to aim so…

Yes, pages of them. Multiple pages of security questions, ciphers used, how data is stored, when is it encrypted, etc. I filled out a 20 pager once. As the company got better and more mature, we had a bunch of canned answers to make it easier and faster....

Entire (excellent) start-ups exist to fill the role of 'RFP library' so that you don't have the whole sales team rewriting the same answers 100 times a year. Loopio saved me hours in the last role I was in that had them - even if you do have to edit some of the responses from colleagues you're not sure passed 9th grade English.

Re: Inside the longest Atlassian outage

#532

Post by kgeist »

drsim wrote: post #442
beachy wrote: post #411

This is extremely poor for a large SaaS company. A standard RFP question for SaaS should be: - Can you restore data for a single customer, and if so, what is the RTO for that operation? A smaller SaaS could be excused for only thinking about full database restores. When you're a scrappy upstart, thinking about hypotheticals is less important than survival. But for any decent size multi-tenanted SaaS, it's imperative…

Which SaaS platforms provide account-level restores? If you contact them and say "please restore our data to as it was last week" those I know do not offer this.

>Which SaaS platforms provide account-level restores?

We restore deleted accounts on request sometimes. There was a client, for example, who forgot to renew the subscription and did nothing for 30 days, so their account was automatically deleted. We restored it from backups. It helps that every tenant has their own isolated database, so it's mostly a matter of restoring that one single DB. Some microservices store data without DB-level sharding, so we have a script which is able to make a partial dump for a specific account.

There's also a popular option to restore deleted data - nothing is ever hard-deleted (it's marked deleted but stays in the DB) and we have a script which can restore individual records (and related records). There's maybe 5 such requests per month.

We don't offer rolling everything back to a specific point in time, though. Technically it's possible by undoing the event queue but it's untested.

We also have a script to migrate customers from cloud to on-premises and back.

Re: Inside the longest Atlassian outage

#533

Post by tinco »

jacksnipe wrote: post #503
buscoquadnary wrote: post #489

Earlier quoted context omitted.

And coders that say all code has bugs are just defeatists that are trying to make excuses for being lazy. Sometimes manure will always hit the fan. Being robust means being able to handle that.

I think this is obviously incorrect. Human error is probabilistic, and the probability of making an error cannot be zero. On the flip side, it’s infeasible to use only provably correct systems; not lazy, but literally not a practical option due to compute costs, developer time, what formal techniques can even be applied to the problem at hand, etc…

It's not about making the probability of an error zero, it's about making sure you can recover from every type of error effectively.

They've underbudgeted for engineering, and they're feeling that now.

Re: Inside the longest Atlassian outage

#534

Post by polote »

kingofpandora wrote: post #458

Engineering mistakes happen. The most inexcusable thing is not communicating with the paying customers who have been affected for over a week. Atlassian's Global Head of Customer Success probably should have been fired but here she is promoting Atlassian Cloud on LinkedIn three days ago: https://www.linkedin.com/mwlite/in/gertie-rizzo-5b70061 Actually reading a bit more, it seems like their customer team was partying…

There is not much csm can do when an outage like that happens more than waiting.

So I don't see why partying at the same time could be an issue. Thats the engineering which made the mistake. So even though they could have communicated better ( and we don't have all the details, we don't know). The true people at fault are Engineering and product leaders.

Engineering mistakes is not an excuse to blame others, and there is difference between a mistake and removing data of production customers

Re: Inside the longest Atlassian outage

#535

Post by [deleted] »

polote wrote: post #534
kingofpandora wrote: post #458

Engineering mistakes happen. The most inexcusable thing is not communicating with the paying customers who have been affected for over a week. Atlassian's Global Head of Customer Success probably should have been fired but here she is promoting Atlassian Cloud on LinkedIn three days ago: https://www.linkedin.com/mwlite/in/gertie-rizzo-5b70061 Actually reading a bit more, it seems like their customer team was partying…

There is not much csm can do when an outage like that happens more than waiting. So I don't see why partying at the same time could be an issue. Thats the engineering which made the mistake. So even though they could have communicated better ( and we don't have all the details, we don't know). The true people at fault are Engineering and product leaders. Engineering mistakes is not an excuse to blame others, and ther…

[deleted]

Re: Inside the longest Atlassian outage

#536

Post by cerved »

polote wrote: post #534
kingofpandora wrote: post #458

Engineering mistakes happen. The most inexcusable thing is not communicating with the paying customers who have been affected for over a week. Atlassian's Global Head of Customer Success probably should have been fired but here she is promoting Atlassian Cloud on LinkedIn three days ago: https://www.linkedin.com/mwlite/in/gertie-rizzo-5b70061 Actually reading a bit more, it seems like their customer team was partying…

There is not much csm can do when an outage like that happens more than waiting. So I don't see why partying at the same time could be an issue. Thats the engineering which made the mistake. So even though they could have communicated better ( and we don't have all the details, we don't know). The true people at fault are Engineering and product leaders. Engineering mistakes is not an excuse to blame others, and ther…

The number one priority of someone maintaining business should be ensuring communication if you want to keep business

Re: Inside the longest Atlassian outage

#537

Post by Karrot_Kream »

syshum wrote: post #480
kingofpandora wrote: post #458

Engineering mistakes happen. The most inexcusable thing is not communicating with the paying customers who have been affected for over a week. Atlassian's Global Head of Customer Success probably should have been fired but here she is promoting Atlassian Cloud on LinkedIn three days ago: https://www.linkedin.com/mwlite/in/gertie-rizzo-5b70061 Actually reading a bit more, it seems like their customer team was partying…

sales never takes the blame. If anyone is fired it will be scapegoats in engineering once they have busted their ass to restore their reward will be the door

Why should sales take the blame when it's engineering's problem? If Sales promised a feature to a customer that was infeasible _then_ it should be Sales problem, but engineers made a mistake so only engineers can clean up the mistake.

Re: Inside the longest Atlassian outage

#538

Post by cerved »

yabones wrote: post #215

What's a good Jira replacement? Redmine? Phabricator? OpenProject? Just leaving the jira server alone and hoping there's no new and exciting zero-days? One thing is clear, these guys are a bunch of cowboys who can't be trusted with any amount of data.

kernel.org

Re: Inside the longest Atlassian outage

#539

Post by josephg »

jacksnipe wrote: post #503
buscoquadnary wrote: post #489

Earlier quoted context omitted.

And coders that say all code has bugs are just defeatists that are trying to make excuses for being lazy. Sometimes manure will always hit the fan. Being robust means being able to handle that.

I think this is obviously incorrect. Human error is probabilistic, and the probability of making an error cannot be zero. On the flip side, it’s infeasible to use only provably correct systems; not lazy, but literally not a practical option due to compute costs, developer time, what formal techniques can even be applied to the problem at hand, etc…

Sure human failure is probabilistic. But you can design around that by stacking reliability-enhancing approaches together.

Let’s say there’s a 10% chance of any given feature being broken. Write a test, (which has another 10% chance of being broken) and now it’s only broken if the test and the code are broken, and broken in the same way. So we’re down to Then add a backup / redundancy system. That has a 10% chance of failure, but if you test it regularly then the backup / restore process only has a 1% chance of failure.

Now we have a system that’s pretty reliable in practice, made out of pieces which are only 90% reliable. And no need for PhD level formal methods.

Just do the obvious robustness steps: Write unit tests. Run them with every commit. Have a backup system. Test it. Have redundant servers. Do stages deployments. Monitor your servers and have an on-call roster. Then when everything is working well, add a chaos monkey to increase the failure rate of all of these parts so your team & software gets practice dealing with problems.

The fact that this bug slipped past all of their reliability engineering - past code review and testing into production and in a way they can’t recover - that smells of sloppy work.

Re: Inside the longest Atlassian outage

#540

Post by opensrcken »

kingofpandora wrote: post #458

Engineering mistakes happen. The most inexcusable thing is not communicating with the paying customers who have been affected for over a week. Atlassian's Global Head of Customer Success probably should have been fired but here she is promoting Atlassian Cloud on LinkedIn three days ago: https://www.linkedin.com/mwlite/in/gertie-rizzo-5b70061 Actually reading a bit more, it seems like their customer team was partying…

Unless you know, with certainty, who is responsible for customer-facing communications , how can you publicly call on a specific person to be fired? Even if you did know, and are correct in your implication, how do you know that she didn't want to do the right thing but was overruled by people above her in the chain? Overall, this is a pretty irresponsible comment, and reflects the kind of mob mentality that I would expect in Reddit, but not HN.

And the bit about Atlassian employees being in Vegas has nothing to do with anything - as if the entire company is supposed to shut down its planned celebration because of an incident that a small subset of the company should be handling.

Post reply on HN