Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Inside the longest Atlassian outage

newsletter.pragmaticengineer.com

Post reply on HN
Print view View original

241–250 of 772 posts

Re: Inside the longest Atlassian outage

#241

Post by csours »

throwaway894345 wrote: post #36
h2odragon wrote: post #4

> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production. Simple concepts aren't that simple at their scale, sure, but I suspect this is skimping details on some truly horrendous monolithic architec…

I would really like to understand who makes the decision to purchase JIRA. It's like the C++ of ticketing software--it does everything because no one wanted to sit down and think critically about the use cases and instead decided it would be easier to say "yes" to every single feature request. It definitely feels like whoever is buying JIRA is not on the team who is using it (maybe IT or finance) because it ticks the…

"I am not a fish" - the people who buy it are not the people who use it. - https://www.ted.com/talks/seth_godin_this_is_broken

Re: Inside the longest Atlassian outage

#242

Post by inopinatus »

seanwilson wrote: post #236
nitinagg wrote: post #2

Selectively restoring data only for certain rows is super hard. But the communications by Atlassian has been the worst I have ever seen in the industry.

> Selectively restoring data only for certain rows is super hard. What's the right way to structure your data here that would make restoring more straightforward here? Is this backup/restore scenario niche or they should have designed for it?

in theory, shard your customer databases 1:1, job done. alas, in practice, many SaaS compromise this two ways:

a) overwhelmed by creeping featuritis, each customer's data has relationships to global tables, and

b) they backup their entire database cluster in one snapshot

and there maybe other gotchas for restoration, like relying on denormalized views and caches that have to be rebuilt. they may also have erroneously assumed that data protection's main value driver is whole-of-system disaster recovery, which can lead to pathologies such as "we don't have a single-customer restoration tool".

this is not a niche scenario

Re: Inside the longest Atlassian outage

#243

Post by SatvikBeri »

throwaway894345 wrote: post #36
h2odragon wrote: post #4

> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production. Simple concepts aren't that simple at their scale, sure, but I suspect this is skimping details on some truly horrendous monolithic architec…

I would really like to understand who makes the decision to purchase JIRA. It's like the C++ of ticketing software--it does everything because no one wanted to sit down and think critically about the use cases and instead decided it would be easier to say "yes" to every single feature request. It definitely feels like whoever is buying JIRA is not on the team who is using it (maybe IT or finance) because it ticks the…

At big companies I've worked at, the justification was that JIRA was the only one that met all the regulatory/compliance requirements. I don't know if this is actually true, but smaller companies certainly don't market compliance as well.

Re: Inside the longest Atlassian outage

#244

Post by ornornor »

mistrial9 wrote: post #225
codycraven wrote: post #185

Earlier quoted context omitted.

It sounds like you've been hurt by the some terrible management practices, I'm truly sorry that some managers think their job is to control their subordinates. However, regarding ticketing systems, in team environments, it is very effective and helpful to have a system that manages the data about the work that has been completed, is being worked, and is planned to be worked on . Part of that system might be defining…

I was told by a lifetime manager turned successful consultant, that roughly fifty percent of engineering firms govern their engineers basically using fear.

> using fear

Could you elaborate? What kind of fear? “You’re fired”? I wonder how effective it actually is because of the current job market and also because I (and others) react very poorly to this kind of tactics: “you want me to fear getting fired? Joke’s on you, please DO fire me, I dare you”

Re: Inside the longest Atlassian outage

#245

Post by bborud »

brianwawok wrote: post #25
R0ger wrote: post #13

I guess this is wake call for the people rushing to SaaS solutions.

Is it? We use JIRA. Not impacted. If this had hit us.. we would just switch to excel or something for a week/month? But maybe we are a very light user of JIRA. Nothing in there can't be replaced. It's "nice" to be able to go look up a 3 year old bug and which client reported it, but not really crucial for day to day ops.

"Switch to Excel for a week/month"

Right.

Re: Inside the longest Atlassian outage

#246

Post by sodality2 »

bombcar wrote: post #235
bluedino wrote: post #145

Regarding the backup restores: I once worked a company that had a data loss issue. There was nothing else we could do, we had exhausted every option we had over almost 40 hours. At the end of the second day, it was decided to restore from backup. We had done this before, as a test. It took about 12 hours to restore the data and another 12 hours to import the data and get back up and running. One small thing was diffe…

In these cases the best thing to do is just give every customer the full month refund; don't make them ask for it.

The best thing to do business-wise, or as a good faith move?

Re: Inside the longest Atlassian outage

#247

Post by notreallyserio »

throwaway894345 wrote: post #36
h2odragon wrote: post #4

> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production. Simple concepts aren't that simple at their scale, sure, but I suspect this is skimping details on some truly horrendous monolithic architec…

I would really like to understand who makes the decision to purchase JIRA. It's like the C++ of ticketing software--it does everything because no one wanted to sit down and think critically about the use cases and instead decided it would be easier to say "yes" to every single feature request. It definitely feels like whoever is buying JIRA is not on the team who is using it (maybe IT or finance) because it ticks the…

JIRA is generally fine software that is good enough for most folks, especially if you're willing to adapt your workflow to it. Where it goes wrong is where tools like Jenkins go wrong: folks add too much customization.

That means the tool is often the wrong one for the job, but instead of picking something that's a better match out of the box folks stick with the easy choice (extend what they have).

Re: Inside the longest Atlassian outage

#248

Post by [deleted] »

yabones wrote: post #215

What's a good Jira replacement? Redmine? Phabricator? OpenProject? Just leaving the jira server alone and hoping there's no new and exciting zero-days? One thing is clear, these guys are a bunch of cowboys who can't be trusted with any amount of data.

[deleted]

Re: Inside the longest Atlassian outage

#249

Post by originalvichy »

yabones wrote: post #215

What's a good Jira replacement? Redmine? Phabricator? OpenProject? Just leaving the jira server alone and hoping there's no new and exciting zero-days? One thing is clear, these guys are a bunch of cowboys who can't be trusted with any amount of data.

For pure engineering teams it’s either Gitlab or Azure Devops. Those are the most common competitors I hear about. If you have non-engineers the choice gets trickier.

Re: Inside the longest Atlassian outage

#250

Post by rwiggins »

femto113 wrote: post #239
tyingq wrote: post #175

Earlier quoted context omitted.

>I still don't understand the strangehold JIRA has on some clients. - Integrations with things like the source code repos, incident management systems, confluence or other wikis, Slack, etc. Moving away from Jira creates a bunch of dead links. - Internal dependence on complex workflows and state transition rules that are implemented in Jira. - Various very customized reports that leaders depend on to make decisions,…

Many years worth of source code filled with comments like // if we don't toggle bit 7 here 10% of transactions will fail on Thursdays // see JIRA issue BIGPROJ-12654 for detailed discussion

When we migrated away from JIRA, we scripted it such that the JIRA issue numbers were recorded in the newly migrated issues exactly because of things like this.
Post reply on HN