Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Inside the longest Atlassian outage

newsletter.pragmaticengineer.com

Post reply on HN
Print view View original

741–750 of 772 posts

Re: Inside the longest Atlassian outage

#741

Post by treis »

inopinatus wrote: post #290
seanwilson wrote: post #277

Earlier quoted context omitted.

> shard your customer databases 1:1 What are the downsides to this?

* makes it much harder to distribute your tables by any other factor, for whatever reason (usually performance, sometimes archival) * disaggregates data that the SaaS might be interested in querying/updating as an aggregate * not all ORM frameworks handle this case well, if at all * dumps are more than a single trivial command basically all your data operations gain an additional dimension of complexity, and you may…

Migrations suck too.

Re: Inside the longest Atlassian outage

#742

Post by treis »

dx034 wrote: post #709
beachy wrote: post #693

Earlier quoted context omitted.

There are lots of downsides to doing absolute partitioning of tenants (along with lots of upsides as you point out). Really annoying things that slash your velocity. You can't easily run pan-customer queries, can't aggregate data. Ironically too using a single database makes full backup and restore much easier.

Wouldn't you extract data anyway to another system for analytics? Running analytics queries on production databases seems a bit risky in any setting?

You can have a read-only replica if you're concerned about performance.

Re: Inside the longest Atlassian outage

#743

Post by treis »

hu3 wrote: post #19
pgwhalen wrote: post #14

Earlier quoted context omitted.

> The fact it's been so long and they still haven't revealed and explained the root cause of the outage They did last night: https://www.atlassian.com/engineering/april-2022-outage-upda...

> Faulty script. Second, the script we used provided both the "mark for deletion" capability used in normal day-to-day operations (where recoverability is desirable), and the "permanently delete" capability that is required to permanently remove data when required for compliance reasons. The script was executed with the wrong execution mode and the wrong list of IDs. The result was that sites for approximately 400 cu…

The bigger story here is that they're restoring data that should have been permanently deleted for compliance reasons.

Re: Inside the longest Atlassian outage

#744

Post by henryfjordan »

txutxu wrote: post #736
Rantenki wrote: post #618

I've repeatedly asked Atlassian if: 1. They can confirm that they have backups of our data (about a thousand stories, substantial confluence, opsgenie history, and three service desks). 2. Will our integrations, configuration, and customizations also be recovered, or will we need to rebuild those once our data is recovered? I have received no response, and no human is even willing to acknowledge those questions. The…

There is a thing that I dont understand, from their blog/report [1] If the script was used in "permanently delete" mode, which is intended for compliance... how do you restore? Is it the only explanation... if the deletion is non-compliant? > Second, the script we used provided both the "mark for deletion" capability used in normal day-to-day operations (where recoverability is desirable), and the "permanently delete…

Under GDPR you don't need to scrub your backups, just your working data:

https://www.itgovernance.eu/blog/en/the-gdpr-how-the-right-t....

Re: Inside the longest Atlassian outage

#745

Post by [deleted] »

lallysingh wrote: post #483
miketria wrote: post #404

Earlier quoted context omitted.

Hi, this is Mike from Atlassian Engineering. You are right the communications from us have not lived up to our standard. We will focus on this specifically once we restore service and get the post incident review out there. More details here: https://www.atlassian.com/engineering/april-2022-outage-upda...

Spamming HN isn't helping your cause man.

[deleted]

Re: Inside the longest Atlassian outage

#746

Post by syshum »

Karrot_Kream wrote: post #547
theteapot wrote: post #543

Earlier quoted context omitted.

What? Because sales interacts with customers and the customer doesn't give a F*$% who's fault it is or even know engineers exist.

This depends on your org structure. Sales does not focus on ongoing relationships at our company, instead we have dedicated account managers who handle relationships with large customers and have a general process to publish status updates during an outage. The outage status update process is guided by an internal outage management team and is a big deal; our C-suite and VPs immediately action on gaps managing status…

In many orgs Sales and customer service is the same team, under the same leadership. However in the context of this conversation we are specifically talking about the "Global Head of Customer Success" if you took the time to read the comment I was reply to. Your reaction (and a few other clues) leads me to believe you are in sales... I am i Right????

In the contest of this outage, the "Global Head of Customer Success" should absolutely be taking the blame for failing to communicate with their customers.

You can attempt to deflect the blame purely onto engineering side of things, as people in sales often to, but you are objectively wrong.

I have suffered long outages with vendors before, the key for me was always COMMUNICATION. They should be communicating with every customer impacted, they should be telling them on going progress, where in the queue they are, etc etc etc.

Trust me as someone that make purchasing choices and recommendations the length of this outage is not the issue, the lack of communications is. As someone that makes purchasing choices and recommendations I do not blame the engineering, I blame sales or "Customer Success" or what ever PR name you want to further deflect sales to be....

Re: Inside the longest Atlassian outage

#747

Post by hw »

function_seven wrote: post #467
bzxcvbn wrote: post #452

Earlier quoted context omitted.

Not every business can afford to go one month without income. What's the best thing for customers? Have the business go bankrupt and irremediably lose access to the service?

It's 400 clients, not all their user base. They can handle the lost income from a small slice of their customers for one month. And if they can't sustain that, then it's even more imperative that those customers migrate away.

400 clients, but how much of their revenue? Were they all small clients? Not to mention longer tail effects of people moving away to competitors even if they werent directly affected?

Re: Inside the longest Atlassian outage

#748

Post by closeparen »

chousuke wrote: post #255
closeparen wrote: post #212

Earlier quoted context omitted.

>ensure the agreed upon process is followed for quality or consistency That is what I mean here by "assembly line" and "control." Making sure that processes lead and individuals follow. Citing consistency as a terminal value in the same breath as quality is also exactly what I mean by the middle-manager aversion to local differences.

Beyond trivial scale, you need good processes so that individuals can do their jobs. If you have no processes, change and development becomes extremely difficult because people will be hunting for documentation all the time, stepping on each other's toes, and making mistakes that they should not be making because they forgot a trivial procedure that was a prerequisite to solving their actual problem. I work with a va…

IMHO the "correct" or at least humane organizational design is that most things happen in local teams, which are of trivial scale and can get along just fine with informal, ad-hoc, or locally varied processes.

Obviously not all work is this way. Sometimes you need to drive a migration that touches every team, and then the technologies of bureaucracy and process become important. But most work should be done in human-scale groups that can be more towards the self-organizing and trust-based end of the spectrum.

However some middle managers take offense to the idea that their different sub-teams have different operating models internally, and lean on technologies like JIRA to try to make them all the same. Middle managers at my company have tried this, not very effectively , so it hasn't hurt me too bad. But I've seen their vision and recoiled in horror.

Re: Inside the longest Atlassian outage

#749

Post by tgmatt »

lloydatkinson wrote: post #148
kgeist wrote: post #142

We use on-premises setups for almost everything (we generally avoid cloud solutions to have full control of our data), sometimes (approximately once a month) it goes down for a few minutes which already feels like a torture because all our processes depend on it, I can't imagine having no access to it for several weeks, all our work would stop to a halt... The office of the guy who administers on-premise servers is l…

Cloud solutions can work well. I've used GitHub, Azure Devops, and BitBucket (another wonderful atlassian product /s) and BitBucket frequently craps out, multiple times a week. We need to rerun builds in TeamCity because BitBucket stops talking to it.

Bitbucket is so slow and unreliable for us we regularly refer to it as Shitbucket. About as useful as a bucket of crap most of the time.

Re: Inside the longest Atlassian outage

#750

Post by sjtindell »

biomcgary wrote: post #287
sjtindell wrote: post #202

Earlier quoted context omitted.

Agreed. This is a problem of scale in my opinion. When we have 10 engineers, it is easy to check in with everyone and know what they are working on and get a status update. When we have 500 engineers, making sure all their tasks are aligning (organizations are one big race condition) is not just hard but impossible without some sort of tracking system. We all want to grow big. To do so, your processes need to change…

Are they unique because their problem domain allows it or because the leadership is uniquely ideologically driven (and competent) to implement efficient, flat systems?

I think it’s mostly the people. They are die hard about their culture. At most of my workplaces, culture is generic and the company would be willing to set whatever culture rules seemed to work.
Post reply on HN