Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Inside the longest Atlassian outage

newsletter.pragmaticengineer.com

Post reply on HN
Print view View original

571–580 of 772 posts

Re: Inside the longest Atlassian outage

#571

Post by mdoms »

wly_cdgr wrote: post #566
opensrcken wrote: post #540

Earlier quoted context omitted.

Unless you know, with certainty, who is responsible for customer-facing communications , how can you publicly call on a specific person to be fired? Even if you did know, and are correct in your implication, how do you know that she didn't want to do the right thing but was overruled by people above her in the chain? Overall, this is a pretty irresponsible comment, and reflects the kind of mob mentality that I would…

They are literally the head of "customer success". The buck should probably not stop ONLY with them, but obviously they're one of the people it needs to stop with

You seem to be implying that customer success is a customer support role. It isn't. Customer success is about helping customers get the greatest business value from your product. They are not going to be munging databases and wrangling backups. Support engineers (among others) do that work.

Re: Inside the longest Atlassian outage

#572

Post by mdoms »

kingofpandora wrote: post #458

Engineering mistakes happen. The most inexcusable thing is not communicating with the paying customers who have been affected for over a week. Atlassian's Global Head of Customer Success probably should have been fired but here she is promoting Atlassian Cloud on LinkedIn three days ago: https://www.linkedin.com/mwlite/in/gertie-rizzo-5b70061 Actually reading a bit more, it seems like their customer team was partying…

A lot of people talking about "firing" in this thread. That's not how Atlassian operates, to their credit. It probably comes from their Australian DNA.

Re: Inside the longest Atlassian outage

#573

Post by brazzledazzle »

iancarroll wrote: post #552
theteapot wrote: post #541

Earlier quoted context omitted.

Not having per tenant backups is sensible? Seems like a bit of an oversight. Doesn't really matter if the hosting is multi-tenant or not.

You can imagine problems restoring one individual tenant's data to an otherwise active database with many tenants; i.e. any cross-tenant primary keys which will have shifted since one tenant's last backup. Separating the backups wouldn't help with the restoration.

It is a complex problem but it’s one worth solving. Just spit balling but I think you could reduce some of the difficulty maintaining it by shifting it away from ops to development. Keep the disaster recovery level database backups for that rainy day but make customer level backup/restores an integrated feature developed and maintained like other services.

Re: Inside the longest Atlassian outage

#574

Post by wly_cdgr »

mdoms wrote: post #571
wly_cdgr wrote: post #566

Earlier quoted context omitted.

They are literally the head of "customer success". The buck should probably not stop ONLY with them, but obviously they're one of the people it needs to stop with

You seem to be implying that customer success is a customer support role. It isn't. Customer success is about helping customers get the greatest business value from your product. They are not going to be munging databases and wrangling backups. Support engineers (among others) do that work.

No post body was provided.

Re: Inside the longest Atlassian outage

#575

Post by galori »

danuker wrote: post #422
bitwise101 wrote: post #297

This talk from Atlassian aged well https://conferences.oreilly.com/software-architecture/sa-eu-...

I am tired of survivor-biased "best practices" advice. I wonder which practices contained there are the worst practices .

This is interesting but can you expand?

My understanding of survivor bias is that you're getting a skewed picture because some of the data was excluded completely.

Re: Inside the longest Atlassian outage

#576

Post by ineedasername »

k8sToGo wrote: post #162
ineedasername wrote: post #152

Something to consider is that Jira can require a great deal of configuration to tailor it to your needs. If you already have a DevOps team of some capacity (not everyone does) then it may only be a small incremental increase to run thinks on prem. I did it myself: I'm ver much not a DevOps person, mostly unfamiliar with optimizing JVM parameters for apps like this, but it still only took me about 5 hours to get thing…

I am Dev Ops, and not Ops. So I try to not waste time with self hosting as much as possible.

DevOps really is not just doing DevOps on cloud platforms and SaaS. Besides, the sysadmin aspects of self hosting should be handled by, well, sysadmin. DevOps should be handling other aspects like developing solutuons necessary to have things (in this case Jira) work together with other systems. (Among other responsibilities) Though DevOps can implemented in different ways with responsibilities that are different from one organization to another, but I've never heard it defined as "we don't deal with on prem"

But I also get the impression that you may just be expressing a preference, not a rule of DevOps? If so then I definitely understand. Custom solutions to integrate or glue disparate systems together is often not the most interesting work. My area... A single word doesn't encompass what I do, I'm a generalist in my domain with one or two specialties, but glueing data together (not the same as a full integration, I know) is a big part of my job, and usually the least interesting.

Though in this case from other comments on prem seems a a dwindling option anyway for Jira. I worked with it about 7 years ago under one of their free licensing programs and disliked it enough that I didn't bother following them after that.

Re: Inside the longest Atlassian outage

#577

Post by brazzledazzle »

boardwaalk wrote: post #456
drsim wrote: post #442

Earlier quoted context omitted.

Which SaaS platforms provide account-level restores? If you contact them and say "please restore our data to as it was last week" those I know do not offer this.

I wouldn’t expect them to advertise such a thing, but the question is “can they recover from their own mistakes” not “can they recover from mine.” I don’t care if this is with an “account-level restore” or whatever; it shouldn’t be my concern.

I’ve seen customer and resource level restores deprioritized more than once and the only hypothetical given serious thought is avoiding helping customers who accidentally deleted something because of the support burden/cost. No one seems to have much concern for what happens when they’re the ones that screwed up.

Re: Inside the longest Atlassian outage

#578

Post by victor9000 »

Wow would you look at that, a complete Atlassian puff piece got published in the WSJ just hours ago.

How peculiar that the biggest active outage in the history of this company is not mentioned once in this "article".

I'm left to assume that PR teams can plant whatever they see fit in the WSJ at a moment's notice. I guess that's what passes for journalism these days.

https://www.wsj.com/articles/atlassian-puts-easy-to-use-codi...

Re: Inside the longest Atlassian outage

#579

Post by anarazel »

nitinagg wrote: post #2

Selectively restoring data only for certain rows is super hard. But the communications by Atlassian has been the worst I have ever seen in the industry.

ISTM the fairly obvious approach would be to bring up a complete copy of the affected database(s) and move the affected tenants to that "copy", while eventually deleting non-affected tenants. Can't imagine they don't have the ability to move tenants to different shards, they got to need that to deal with quickly growing customers etc.
Post reply on HN