Live data from Hacker News

Inside the longest Atlassian outage

newsletter.pragmaticengineer.com

671–680 of 772 posts

Re: Inside the longest Atlassian outage

#671

Earlier quoted context omitted.

> Atlassian - MUST DO BETTER. it’s not like people will stop using jira and confluence, lol they basically have a monopoly there

Nobody likes using this stuff though. I think the new github boards might give jira a run for their money given the cost... free.

Here is someone that likes it, and is usually put down by lack of features in competing products.

Re: Inside the longest Atlassian outage

#672

Earlier quoted context omitted.

I don't have Postgres/MySQL scaling experience beyond wrangling some largeish monoliths with But, You can imagine problems restoring one individual tenant's data to an otherwise active database with many tenants any cross-tenant primary keys Why would multiple tenants share a database? Sharing a database server, yes, but sharing databases and mingling primary keys and such? That's such a recipe for disaster; giving e…

Couple of scenarios come to mind... 1) Its super common even in multitenant systems to have a common database with configuration information (for example) which serves all tenants, and tenant-specific databases used alongside that to host their private data. 2) Back when sharding started to be a popular scaling pattern, tenants were not always split up by the tenant boundary but by some other reliable key. Obviously…

    1) Its super common even in multitenant 
    systems to have a common database with 
    configuration information (for example) 
    which serves all tenants, and tenant-specific 
    databases used alongside that to host their 
    private data.
Yeah, for sure. This is definitely what I'd expect to see, but I would also expect that to make individual client restores pretty easy, assuming the individual client backups themselves weren't trashed.

One wouldn't imagine that the shared config database would have a dependency on any of the individual client databases and that they could therefore be moved/dropped/restored at will, independently of the shared config database.

    2) Back when sharding started to be a popular 
    scaling pattern, tenants were not always split 
    up by the tenant boundary but by some other 
    reliable key. 
I guess that makes sense. I mean, after all, it does allow large/demanding clients to span multiple databases I guess.

Re: Inside the longest Atlassian outage

#673
post #133

The sad truth is that with 99.8% of customers unaffected, it was probably thought to be a minor issue. If those customers didn't have Gergely's ear we probably wouldn't have heard about it.

Hi, this is Mike from Atlassian Engineering. Not a minor issue. Once we knew the extent and severity of the incident, we had hundreds of engineers engaged and working to restore service.

I should have clarified, that I was talking about leadership's external communication on the incident, like in the article. Nobody doubted you were working around the clock, or with lots of people involved.

Re: Inside the longest Atlassian outage

#674
post #663

Earlier quoted context omitted.

I was down, my instance is fully restored right now. 1. They do, every 25 hours, via snapshot. I have spoked to their team since the incident and that same thing is in this article. 2. Yes, they recover all of it. Some things have had issues, external mailboxes attached to service management projects, some attachment rendering slowness. Filters needing to be overlayed into our instance again, but otherwise it is runn…

Life saving Jira tickets?

FWIW Jira and Confluence and used in life sciences firms, though they are rarely more useful than Google Docs. Self hosting in this context was the norm but that is changing.

Re: Inside the longest Atlassian outage

#675
post #509

Earlier quoted context omitted.

Yes, pages of them. Multiple pages of security questions, ciphers used, how data is stored, when is it encrypted, etc. I filled out a 20 pager once. As the company got better and more mature, we had a bunch of canned answers to make it easier and faster....

Entire (excellent) start-ups exist to fill the role of 'RFP library' so that you don't have the whole sales team rewriting the same answers 100 times a year. Loopio saved me hours in the last role I was in that had them - even if you do have to edit some of the responses from colleagues you're not sure passed 9th grade English.

Any other startups you can recommend? I'm filling in my first RFP in a decade and answering what they mean to ask with the questions rather than answering questions literally is not something that comes easily to me.

Plus coming up with an answer to the vague question on "describe your project methodology" (I build what you want, it works - nope, they expect half a page). Or the 3 questions on project management systems and communication software choices that to my reading should have the same answer.

Re: Inside the longest Atlassian outage

#676
post #480

Earlier quoted context omitted.

sales never takes the blame. If anyone is fired it will be scapegoats in engineering once they have busted their ass to restore their reward will be the door

Reformed salesperson here: bullshit. Atlassian famously eschews the exact sales teams whose job it would be to manage direct customer comms in an outage like this one, and to be the lightning rod for the understandable customer frustration. In the past, I've been the guy that gets the angry text message from the customer and has to carefully paper over the gaps in communication from higher ups. It's not fun being the…

They also seem to eschew a proper UX design team, quality assurance and even proper engineering, judging how "well" their software works.

Re: Inside the longest Atlassian outage

#677
post #668

Earlier quoted context omitted.

Well right now customers are getting zero value. It’s a bad look to be partying during an all hands on deck emergency.

More hands on that deck probably don't help.

And yet maybe they could have personally responded to the impacted customers with informative, timely updates instead of waiting several days and then sending automated replies from templates.

Re: Inside the longest Atlassian outage

#678

Earlier quoted context omitted.

I would really like to understand who makes the decision to purchase JIRA. It's like the C++ of ticketing software--it does everything because no one wanted to sit down and think critically about the use cases and instead decided it would be easier to say "yes" to every single feature request. It definitely feels like whoever is buying JIRA is not on the team who is using it (maybe IT or finance) because it ticks the…

My relatively small team at a massive enterprise built all our report generation tools around JIRA for an entire class of offerings. It's been easier for them to justify continuing to pay for JIRA and keep it propped up than to develop (or migrate to) a new solution. As the lone dev on the team I've been continually astounded by my leadership's willingness to commit more and more to tech debt laden paths. The notion…

I once worked for a company that did the same - but with Lotus Notes, in 2013. Modified it into a full-fledged ticketing- and time-tracking tool. Using it took a half hour out of each workday.

Re: Inside the longest Atlassian outage

#679

Earlier quoted context omitted.

> Atlassian - MUST DO BETTER. it’s not like people will stop using jira and confluence, lol they basically have a monopoly there

Nobody likes using this stuff though. I think the new github boards might give jira a run for their money given the cost... free.

My engineering team runs off Github Issues > Projects > Project Boards but it's been hard to get anyone outside of product managers and devs themselves to be fully immersed in it. Sales and management throw their hands up in the air and say 'Github Issues is too hard'. I'm a PM so I've fought and won this battle to stay on Github Issues because I could quantify that a 5-10% loss in dev productivity in moving to a 'management friendly' tool would equal so many thousands of dollars of developer hours per month.

Re: Inside the longest Atlassian outage

#680
post #621

Earlier quoted context omitted.

I don't have a ton of experience doing this, but with a lot of multitenant you just give each tenant their own database in the first place. It solves multiple classes of problems. On a single Postgres instance you can (at least theoretically) have 4 billion databases per instance.

Most of the multi-tenant SaaS products I've worked with do NOT have per tenant databases. I'm sure some do, but the bulk of the multi-tenant products use one (or several) larger databases.

> I'm sure some do, but the bulk of the multi-tenant products use one (or several) larger databases.

In that case, the tradeoff between isolation and ease of development is made. That said, having a schema per user (even if in the same physical database) seems like a nice approach, if you can stomach the overhead and added ops complexity.

Post reply on HN