Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Inside the longest Atlassian outage

newsletter.pragmaticengineer.com

Post reply on HN
Print view View original

291–300 of 772 posts

Re: Inside the longest Atlassian outage

#291

Post by dahdum »

radicaldreamer wrote: post #75
1970-01-01 wrote: post #63

Interesting note: Atlassian stock (NASDAQ: TEAM) is up 4% as of noon today.

It might be a good short opportunity... I imagine a lot of customers are kicking off their own internal process for migrating away from JIRA. By the time they actually do, it'll be at least a couple of quarters from now, which is when the customer hit will start materializing in quarterly results for the company. Maybe time to throw a few chips at some long term puts?

I wouldn’t short. They just slapped 400+ customers and likely hundreds of thousands of users in the face and the C-suite didn’t think it was important to even acknowledge.

That might look like incompetence, but I think it’s confidence. They know the switching costs for large orgs are so high they can treat these people like trash and few if any will leave. I wouldn’t be surprised if the total number of seats among affected customers has gone up in a few months. By failing to acknowledge the problem they’ve kept it out of the mainstream media and financial press.

They have their customers by the balls and don’t respect them. That’s a short term bullish signal to me.

Re: Inside the longest Atlassian outage

#292

Post by mh- »

escot wrote: post #284

When doing bulk deletes like this what safe guards do you put in place, other than testing the script up/down in another environment, turning off app servers etc (which Im guessing they did not do)?

Depends how complex the query/procedure is.

Naive approach, replace delete with select and see if you're surprised at the results.

More mature approach, especially in an environment where engineers are running bulk changes against the database, you don't do bulk deletes. You change that delete into an update that marks things for later collection.

One tactic I've seen that worked, assuming you have straightforward relational tables: you add a "marked for deletion" column whose value is an identifier for the single run of the bulk job you just did. Then you can query rows with that value in that column to ensure it had the desired effect. If you're satisfied, you run another bulk job which doesn't re-run your original query.. it just deletes rows with that "marked" value.

Lots of places rely on schema-enforced foreign keys and cascading deletes though. In that case, my recommendation is: don't.

Re: Inside the longest Atlassian outage

#293

Post by baq »

chousuke wrote: post #255
closeparen wrote: post #212

Earlier quoted context omitted.

>ensure the agreed upon process is followed for quality or consistency That is what I mean here by "assembly line" and "control." Making sure that processes lead and individuals follow. Citing consistency as a terminal value in the same breath as quality is also exactly what I mean by the middle-manager aversion to local differences.

Beyond trivial scale, you need good processes so that individuals can do their jobs. If you have no processes, change and development becomes extremely difficult because people will be hunting for documentation all the time, stepping on each other's toes, and making mistakes that they should not be making because they forgot a trivial procedure that was a prerequisite to solving their actual problem. I work with a va…

To add to that, and perhaps educate your downvoters a bit, it can be very hard to imagine why or when such strict processes are helpful without having direct experience with organizations of sufficient scale. It literally boggles the mind but the process truly is king when there are hundreds (or thousands) of individuals working on a single product.

Re: Inside the longest Atlassian outage

#295

Post by bigtones »

ksala_ wrote: post #122
h2odragon wrote: post #4

> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production. Simple concepts aren't that simple at their scale, sure, but I suspect this is skimping details on some truly horrendous monolithic architec…

Personally, given the multi-day outage, I think I would just restore everything to a separate system, and then only point the affected customers to this new system. Take the hit of having two fully separated system initially, and work on reconciling them after without having to worry about the on-going outage. I wonder if they're not doing this due to some tech limitations, to avoid taking the financial cost of runni…

That's a really good idea.

Re: Inside the longest Atlassian outage

#296

Post by robertlagrant »

malermeister wrote: post #229
codycraven wrote: post #185

Earlier quoted context omitted.

It sounds like you've been hurt by the some terrible management practices, I'm truly sorry that some managers think their job is to control their subordinates. However, regarding ticketing systems, in team environments, it is very effective and helpful to have a system that manages the data about the work that has been completed, is being worked, and is planned to be worked on . Part of that system might be defining…

> ensure the agreed upon process is followed for quality or consistency. Isn't that just a more corporate way of phrasing "control"?

Not in a negative way. You want to trust engineers to always have changes built and tested before they go to production, but when something egregious happens you need to go back and see what went wrong. You can choose to interpret that as control, but really the only alternative (often cited) is "Well that shouldn't ever happen, so you don't need tooling to support that situation".

And that is not a useful way of thinking when you have real engineers writing software that people depend on.

Re: Inside the longest Atlassian outage

#298

Post by theptip »

closeparen wrote: post #136
throwaway894345 wrote: post #36

Earlier quoted context omitted.

I would really like to understand who makes the decision to purchase JIRA. It's like the C++ of ticketing software--it does everything because no one wanted to sit down and think critically about the use cases and instead decided it would be easier to say "yes" to every single feature request. It definitely feels like whoever is buying JIRA is not on the team who is using it (maybe IT or finance) because it ticks the…

JIRA is a framework for making assembly lines out of knowledge workers. When you're a middle manager at a decent sized company, a major problem you face is that the mass of knowledge workers beneath you are opaque : you have no way of knowing whether they're working or not. Another problem you face is that they're uppity : people who went to college and got used to managing their own time now have all kinds of idiosy…

I think you've got part of the answer here, but are selling it short. Jira is the most complex task-processing rule engine that is also easy enough for a small team to operate, and also has the broadest set of integrated tools of any offering.

You can use Jira as a simple Scrum board, a Kanban board, or you can build enforced-process monstrosities. You can build customer-support / internal-helpdesk workflows, or even model internal work-item-oriented business processes, etc. Now, as you point out, just because you can doesn't mean you should, and many orgs fall into the trap of making issue workflows overly-restrictive. But most companies (I believe) choose Jira before they choose those hairy task workflows. Startups with zero process use Jira.

Also, you can integrate it all together to give good-enough dashboards/roadmaps, good-enough (for some, not me) docs integrations with Confluence, Git integration with Bitbucket etc. -- while there are big issues with these systems, I think it would be myopic to ignore the real benefits of working in one integrated stack where every design doc you write has dynamically-updated labels and auto-complete for each issue you type in.

For context, I use Jira for tasks and don't love it, found Confluence to be really annoying and so I don't use it, and prefer Gitlab to Bitbucket, but I think you have to recognize these unique selling points. If all Jira had to offer was the rule engine it would not be as widely used.

Re: Inside the longest Atlassian outage

#300

Post by brimble »

user22 wrote: post #262
brimble wrote: post #107

Earlier quoted context omitted.

If they don't lose data, two weeks of downtime every few years might be cheaper than the cost of switching. Plus, it's not like you know the thing you switch to will be any better, if it's another SaaS.

Let's say we have an announced release schedule on may 1st. With the tools down, there is no way to meet that date. For a 4 billion dollar company, this can make a huge difference in revenue. For a public company, the stock will definitely drop when it's announced the revenue goals were missed because the tools were down. For companies of size, the cost of tools being down for 3 weeks can easily be in the multi-milli…

Again, part of the trouble is it's hard to gain enough certainty that the thing you switch to—self-hosted, or another service—won't be at least as bad. You can look at their past record, but then, when's the last time Atlassian had this happen? (or maybe they've been having similar issues every year or two and I've just not noticed, in which case, yeah, it's probably a safe bet that switching to almost anything else would be an improvement)
Post reply on HN