Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Inside the longest Atlassian outage

newsletter.pragmaticengineer.com

Post reply on HN
Print view View original

71–80 of 772 posts

Re: Inside the longest Atlassian outage

#71

Post by dylan604 »

TheJoeMan wrote: post #39
hu3 wrote: post #19

Earlier quoted context omitted.

> Faulty script. Second, the script we used provided both the "mark for deletion" capability used in normal day-to-day operations (where recoverability is desirable), and the "permanently delete" capability that is required to permanently remove data when required for compliance reasons. The script was executed with the wrong execution mode and the wrong list of IDs. The result was that sites for approximately 400 cu…

I suppose that’s why you don’t combine a tazer and gun into 1 device with 2 triggers.

Instead, you make it with one trigger and a PRNG that decides which gets activated. Just hope you've chosen the right PRNG!!

Re: Inside the longest Atlassian outage

#72

Post by jasd »

nemothekid wrote: post #45

> Most of them said they won’t leave the Atlassian stack, as long as they don’t lose data. This is because moving is complex and they don’t see a move would mitigate a risk of a cloud provider going down. I still don't understand the strangehold JIRA has on some clients. I can't quickly think of another SaaS product that could be down for almost 2 weeks and not have most customers leave.

Even if they don't, I imagine they will have conversations internally to see what's feasible. It's just really difficult for an organization to move away from a product that everyone has learnt how to use. The company I work for is struggling to move away from something as simple as a collaborative editor, when I feel like I find no difference between the two products.

Re: Inside the longest Atlassian outage

#73

Post by tenacious_tuna »

throwaway894345 wrote: post #36
h2odragon wrote: post #4

> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production. Simple concepts aren't that simple at their scale, sure, but I suspect this is skimping details on some truly horrendous monolithic architec…

I would really like to understand who makes the decision to purchase JIRA. It's like the C++ of ticketing software--it does everything because no one wanted to sit down and think critically about the use cases and instead decided it would be easier to say "yes" to every single feature request. It definitely feels like whoever is buying JIRA is not on the team who is using it (maybe IT or finance) because it ticks the…

My relatively small team at a massive enterprise built all our report generation tools around JIRA for an entire class of offerings. It's been easier for them to justify continuing to pay for JIRA and keep it propped up than to develop (or migrate to) a new solution.

As the lone dev on the team I've been continually astounded by my leadership's willingness to commit more and more to tech debt laden paths. The notion that all software requires maintenance is anathema to them, and it's led us to be 'cornered' into decisions re: what software we can use / where we can invest our discretionary funding.

Moreover, we're constrained by the parent mega-enterprise's software purchase policies; JIRA's already approved (and run elsewhere in the enterprise), whereas off-the-shelf or SaaSy alternatives are significantly harder to get buy-in for. (No using corporate cards for SaaS, all purchases need to go through the quote/purchase-order process, etc).

Re: Inside the longest Atlassian outage

#74

Post by msh »

dabeeeenster wrote: post #33
radicaldreamer wrote: post #28

Earlier quoted context omitted.

I can't believe that they would intermix the data in that way... but if they did, godspeed to them, they're likely still overpromising what can be done in this time frame.

How else do you run a multitenancy platform?

At minimum separate tables for each tenant.

Re: Inside the longest Atlassian outage

#75

Post by radicaldreamer »

1970-01-01 wrote: post #63

Interesting note: Atlassian stock (NASDAQ: TEAM) is up 4% as of noon today.

It might be a good short opportunity... I imagine a lot of customers are kicking off their own internal process for migrating away from JIRA. By the time they actually do, it'll be at least a couple of quarters from now, which is when the customer hit will start materializing in quarterly results for the company.

Maybe time to throw a few chips at some long term puts?

Re: Inside the longest Atlassian outage

#76

Post by a4isms »

RandallBrown wrote: post #50
throwaway894345 wrote: post #36

Earlier quoted context omitted.

I would really like to understand who makes the decision to purchase JIRA. It's like the C++ of ticketing software--it does everything because no one wanted to sit down and think critically about the use cases and instead decided it would be easier to say "yes" to every single feature request. It definitely feels like whoever is buying JIRA is not on the team who is using it (maybe IT or finance) because it ticks the…

I think people buy JIRA because you can set it up however you want. I've seen it almost as simple as Trello and much more complicated. It doesn't have to be terrible, it just usually is. If JIRA didn't allow you to make it terrible, it wouldn't allow for some of the absurd things that people want it for and those companies might not buy it.

They used to say of Microsoft Word, "Nobody uses more than 5% of its features, but every company uses a different 5%."

The saying is apocryphal and unlikely to be accurate, but the shape of the thing its describing applies to almost every piece of enterprise software whether installed on-prem or SaaS.

And as another comment points out, at Enterprise scale you can substitute "team" or "group" for customer. Every team might use a different 5%, and unless you standardize their processes, you have to buy the product that can accomodate all of their needs.

Re: Inside the longest Atlassian outage

#77

Post by Forgeties79 »

throwaway894345 wrote: post #36
h2odragon wrote: post #4

> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production. Simple concepts aren't that simple at their scale, sure, but I suspect this is skimping details on some truly horrendous monolithic architec…

I would really like to understand who makes the decision to purchase JIRA. It's like the C++ of ticketing software--it does everything because no one wanted to sit down and think critically about the use cases and instead decided it would be easier to say "yes" to every single feature request. It definitely feels like whoever is buying JIRA is not on the team who is using it (maybe IT or finance) because it ticks the…

I really, really like Trello and am dreading the day when atlassian starts tinkering with it in any real capacity. As a content creator, it is the first workflow system I’ve ever seen that I can effectively share with my client. It’s so simple and streamlined and the fact that I’ve stuck with it despite my ADHD says a lot.

Clients add their notes to the card, I check the boxes as I hit the notes, and I move the card further right as we enter different stages of the post production process. We then have a column of every completed project, which is incredibly easy to sift through if we need to revisit something. It’s literally left to right in the workflow, it visually is telling me where we are at all times.

It’s incredibly simple and elegant. For fast turnaround, relatively stripped down content (like podcasts) there is nothing like it.

Re: Inside the longest Atlassian outage

#78

Post by systemvoltage »

throwaway894345 wrote: post #36
h2odragon wrote: post #4

> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production. Simple concepts aren't that simple at their scale, sure, but I suspect this is skimping details on some truly horrendous monolithic architec…

I would really like to understand who makes the decision to purchase JIRA. It's like the C++ of ticketing software--it does everything because no one wanted to sit down and think critically about the use cases and instead decided it would be easier to say "yes" to every single feature request. It definitely feels like whoever is buying JIRA is not on the team who is using it (maybe IT or finance) because it ticks the…

What's wrong with C++? Seems unfair to compare it with JIRA.

Re: Inside the longest Atlassian outage

#79

Post by dylan604 »

notdang wrote: post #70
mkl95 wrote: post #23

Earlier quoted context omitted.

> Communication gap. First, there was a communication gap between the team that requested the deactivation and the team that ran the deactivation. Instead of providing the IDs of the intended app being marked for deactivation, the team provided the IDs of the entire cloud site where the apps were to be deactivated. So what they are saying is that they are not testing scripts at some staging server before running them…

it seems that it worked as intended, thus they have a QA process. The problem was in the wrong IDs provided and I doubt that at their scale they have a staging environment that duplicates the customer data.

Would it be bad practice to append values to a GUID type of ID that would help a human recognize them? For instance, in this specific case they wanted app IDs as APP-XXXXX-XXXX-blahblah and CLOUD-XXXXX-blahblah.

I'm not looking to help their specific problems, but this is more from a general question I've thought of doing but never have done just because I'm sure I'd get laughed at for blazing my own trail

Re: Inside the longest Atlassian outage

#80

Post by eli »

radicaldreamer wrote: post #75
1970-01-01 wrote: post #63

Interesting note: Atlassian stock (NASDAQ: TEAM) is up 4% as of noon today.

It might be a good short opportunity... I imagine a lot of customers are kicking off their own internal process for migrating away from JIRA. By the time they actually do, it'll be at least a couple of quarters from now, which is when the customer hit will start materializing in quarterly results for the company. Maybe time to throw a few chips at some long term puts?

Aren't most customers in 12+ month contracts? A migration seems like it would take many months to select a new vendor and migrate regardless. Be careful about the date on those puts. It's pretty hard to out-think the market on this kind of stuff. I'd just as soon bet the other way: few customers will actually churn and in 6 months this won't really matter.
Post reply on HN