Earlier quoted context omitted.
> Atlassian - MUST DO BETTER. it’s not like people will stop using jira and confluence, lol they basically have a monopoly there
Nobody likes using this stuff though. I think the new github boards might give jira a run for their money given the cost... free.
Inside the longest Atlassian outage
671–680 of 772 posts
Re: Inside the longest Atlassian outage
#672Earlier quoted context omitted.
I don't have Postgres/MySQL scaling experience beyond wrangling some largeish monoliths with But, You can imagine problems restoring one individual tenant's data to an otherwise active database with many tenants any cross-tenant primary keys Why would multiple tenants share a database? Sharing a database server, yes, but sharing databases and mingling primary keys and such? That's such a recipe for disaster; giving e…
Couple of scenarios come to mind... 1) Its super common even in multitenant systems to have a common database with configuration information (for example) which serves all tenants, and tenant-specific databases used alongside that to host their private data. 2) Back when sharding started to be a popular scaling pattern, tenants were not always split up by the tenant boundary but by some other reliable key. Obviously…
1) Its super common even in multitenant
systems to have a common database with
configuration information (for example)
which serves all tenants, and tenant-specific
databases used alongside that to host their
private data.
Yeah, for sure. This is definitely what I'd expect to see, but I would also expect that to make individual client restores pretty easy, assuming the individual client backups themselves weren't trashed.One wouldn't imagine that the shared config database would have a dependency on any of the individual client databases and that they could therefore be moved/dropped/restored at will, independently of the shared config database.
2) Back when sharding started to be a popular
scaling pattern, tenants were not always split
up by the tenant boundary but by some other
reliable key.
I guess that makes sense. I mean, after all, it does allow large/demanding clients to span multiple databases I guess.Re: Inside the longest Atlassian outage
#673The sad truth is that with 99.8% of customers unaffected, it was probably thought to be a minor issue. If those customers didn't have Gergely's ear we probably wouldn't have heard about it.
Hi, this is Mike from Atlassian Engineering. Not a minor issue. Once we knew the extent and severity of the incident, we had hundreds of engineers engaged and working to restore service.
Re: Inside the longest Atlassian outage
#674Earlier quoted context omitted.
I was down, my instance is fully restored right now. 1. They do, every 25 hours, via snapshot. I have spoked to their team since the incident and that same thing is in this article. 2. Yes, they recover all of it. Some things have had issues, external mailboxes attached to service management projects, some attachment rendering slowness. Filters needing to be overlayed into our instance again, but otherwise it is runn…
Life saving Jira tickets?
Re: Inside the longest Atlassian outage
#675Earlier quoted context omitted.
Yes, pages of them. Multiple pages of security questions, ciphers used, how data is stored, when is it encrypted, etc. I filled out a 20 pager once. As the company got better and more mature, we had a bunch of canned answers to make it easier and faster....
Entire (excellent) start-ups exist to fill the role of 'RFP library' so that you don't have the whole sales team rewriting the same answers 100 times a year. Loopio saved me hours in the last role I was in that had them - even if you do have to edit some of the responses from colleagues you're not sure passed 9th grade English.
Plus coming up with an answer to the vague question on "describe your project methodology" (I build what you want, it works - nope, they expect half a page). Or the 3 questions on project management systems and communication software choices that to my reading should have the same answer.
Re: Inside the longest Atlassian outage
#676Earlier quoted context omitted.
sales never takes the blame. If anyone is fired it will be scapegoats in engineering once they have busted their ass to restore their reward will be the door
Reformed salesperson here: bullshit. Atlassian famously eschews the exact sales teams whose job it would be to manage direct customer comms in an outage like this one, and to be the lightning rod for the understandable customer frustration. In the past, I've been the guy that gets the angry text message from the customer and has to carefully paper over the gaps in communication from higher ups. It's not fun being the…
Re: Inside the longest Atlassian outage
#677Earlier quoted context omitted.
Well right now customers are getting zero value. It’s a bad look to be partying during an all hands on deck emergency.
More hands on that deck probably don't help.
Re: Inside the longest Atlassian outage
#678Earlier quoted context omitted.
I would really like to understand who makes the decision to purchase JIRA. It's like the C++ of ticketing software--it does everything because no one wanted to sit down and think critically about the use cases and instead decided it would be easier to say "yes" to every single feature request. It definitely feels like whoever is buying JIRA is not on the team who is using it (maybe IT or finance) because it ticks the…
My relatively small team at a massive enterprise built all our report generation tools around JIRA for an entire class of offerings. It's been easier for them to justify continuing to pay for JIRA and keep it propped up than to develop (or migrate to) a new solution. As the lone dev on the team I've been continually astounded by my leadership's willingness to commit more and more to tech debt laden paths. The notion…
Re: Inside the longest Atlassian outage
#679Earlier quoted context omitted.
> Atlassian - MUST DO BETTER. it’s not like people will stop using jira and confluence, lol they basically have a monopoly there
Nobody likes using this stuff though. I think the new github boards might give jira a run for their money given the cost... free.
Re: Inside the longest Atlassian outage
#680Earlier quoted context omitted.
I don't have a ton of experience doing this, but with a lot of multitenant you just give each tenant their own database in the first place. It solves multiple classes of problems. On a single Postgres instance you can (at least theoretically) have 4 billion databases per instance.
Most of the multi-tenant SaaS products I've worked with do NOT have per tenant databases. I'm sure some do, but the bulk of the multi-tenant products use one (or several) larger databases.
In that case, the tradeoff between isolation and ease of development is made. That said, having a schema per user (even if in the same physical database) seems like a nice approach, if you can stomach the overhead and added ops complexity.