Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Inside the longest Atlassian outage

newsletter.pragmaticengineer.com

Post reply on HN
Print view View original

641–650 of 772 posts

Re: Inside the longest Atlassian outage

#641

Post by throwaway892238 »

galori wrote: post #575
danuker wrote: post #422

Earlier quoted context omitted.

I am tired of survivor-biased "best practices" advice. I wonder which practices contained there are the worst practices .

This is interesting but can you expand? My understanding of survivor bias is that you're getting a skewed picture because some of the data was excluded completely.

In this case it's survivor bias in that "We did these things and we didn't fail, ergo these things must be great."

Whenever you see a talk like this, always assume that it's BS. It might not be used by any real customers, or might still be in development. There might be a bunch of fires happening all the time due to things the talk doesn't mention. And it might be shuttered the next month if it's too expensive, complicated, obscure, or hard to support. These talks should only be considered aspirational sources of ideas, but never taken as a gold-standard battle-tested model, until they tell you how it fails. Only after you know how a system fails and how to respond to it can it be said to be reliable.

Re: Inside the longest Atlassian outage

#642

Post by wfme »

fortran77 wrote: post #623
wfme wrote: post #606

Earlier quoted context omitted.

Completely agree. Can we please keep the pitchforks on Reddit. It's so incredibly easily to point the finger when you really have no idea. I'm all for calling out their lack of communication, but picking someone you deem to be responsible and calling for their head is pretty medieval.

Why do you think "pitchforks" are on or should be on Reddit? That's a very odd thing to say on Hacker News. And if you don't like Reddit, just don't go there!

At no point did I say anything _should_ be on Reddit - I have no influence or say over what goes on on Reddit, but if you take a look through many recent posts, you will find massive amounts of mob mentality. Thankfully this kind of behaviour is much less common on hn and the community is generally much more open to discussion. I would love for this to continue to be so.

Re: Inside the longest Atlassian outage

#643

Post by Sebguer »

staticautomatic wrote: post #622
PragmaticPulp wrote: post #611

Earlier quoted context omitted.

> Actually reading a bit more, it seems like their customer team was partying in Las Vegas instead of taking care of business: https://www.linkedin.com/mwlite/feed/hashtag/atlassianteam22 This was a conference they hosted , not just some Atlassian team members partying in Vegas: https://events.atlassian.com/team22 Thousands of Atlassian customers bought tickets, flights, and hotels for the event. It's unreasonable to…

Fair enough, but anyone who’s been around the block on crisis communications knows that you’re supposed to hit the big red “stop all regular communications until we get a fucking handle on this” button.

You think Atlassian should have the ability to stop posts on all of their individual employees' social media posts?

Re: Inside the longest Atlassian outage

#644

Post by mh- »

mdoms wrote: post #567
mh- wrote: post #261

Earlier quoted context omitted.

That makes this prolonged restore process all the more confusing, then. I (and many others) assumed they had to graft in data from backups since a full restore would clobber newer changes from unaffected customers. If they're all isolated in their own logical per-tenant DBs, I'm really at a loss for what is making restoration take 3 weeks for 400 tenants. I understand if you'd rather not venture into it, but care to…

To be honest I'm at a bit of a loss too. My speculation is that since they went all-in on microservices and utilizing various AWS services (something that was underway when I worked there) their data stores have become very much more disparate. For example, they have the main PostgreSQL data store. Surely that's easy to restore. But the users in that DB have a "foreign key" (in a logical sense, not physical) to the I…

Thanks for the insights. It does seem to have, ahem, decomposed.

Re: Inside the longest Atlassian outage

#645

Post by mb7733 »

krinchan wrote: post #153
chupchap wrote: post #57

Earlier quoted context omitted.

A lot of companies have integrations to atlassian suite which might not be easy to shift from. Secondly, there are a lot of individual competitors to Jira, Confluence and Bitbucket but which competitor can offer all three under a single invoice? May be Microsoft, can't think of anyone else. Also for such an extended downtime the customers are entitled to a discount or a credit note which a lot of CXOs consider in the…

We are in a similar place with Slack. We moved from HipChat to Slack and that was painful enough. Then the company noticed we get Teams for "free" and they tried to push us over to it. But folks have so much automation (because "ChatOps" is that new new) that is pushing things into Slack the company eventually gave up.

At least in that case, Slack is the better option. While using Jira is a worst-case scenario.

Re: Inside the longest Atlassian outage

#646

Post by Mysterise »

lallysingh wrote: post #483
miketria wrote: post #404

Earlier quoted context omitted.

Hi, this is Mike from Atlassian Engineering. You are right the communications from us have not lived up to our standard. We will focus on this specifically once we restore service and get the post incident review out there. More details here: https://www.atlassian.com/engineering/april-2022-outage-upda...

Spamming HN isn't helping your cause man.

There is irony in complaining about over-communication when it's in response to criticisms of under-communication.

Re: Inside the longest Atlassian outage

#647

Post by galangalalgol »

Karrot_Kream wrote: post #545
Animats wrote: post #472

Earlier quoted context omitted.

Rather, customers must stop using Atlassian cloud services.

This is a great opportunity to be an Atlassian competitor. I'm sure more than one business added Atlassian cloud services as a business risk in the meantime, even if they weren't affected.

gitlab seems to be eating their lunch already. Its bit bucket plus Jenkins plus the best bits of jira and it's starting price is free... I like confluence a lot, but separate design/documentation tools with their files pushed up to the repo are good enough. Latex, doxygen, or PowerPoint, whatever you want.

Re: Inside the longest Atlassian outage

#648

Post by karmajunkie »

JohnBooty wrote: post #609
iancarroll wrote: post #552

Earlier quoted context omitted.

You can imagine problems restoring one individual tenant's data to an otherwise active database with many tenants; i.e. any cross-tenant primary keys which will have shifted since one tenant's last backup. Separating the backups wouldn't help with the restoration.

I don't have Postgres/MySQL scaling experience beyond wrangling some largeish monoliths with But, You can imagine problems restoring one individual tenant's data to an otherwise active database with many tenants any cross-tenant primary keys Why would multiple tenants share a database? Sharing a database server, yes, but sharing databases and mingling primary keys and such? That's such a recipe for disaster; giving e…

Couple of scenarios come to mind...

1) Its super common even in multitenant systems to have a common database with configuration information (for example) which serves all tenants, and tenant-specific databases used alongside that to host their private data.

2) Back when sharding started to be a popular scaling pattern, tenants were not always split up by the tenant boundary but by some other reliable key. Obviously this isn't true multitenancy and I think most DBAs would discourage the pattern today. However, given the age of the products at Atlassian (and assuming a fast-and-loose engineering culture, which has been alluded to elsewhere) its entirely possible that parts of these products as well as the entire product itself may use this kind of sharding.

Bottom line, we can only hypothesize unless and until someone from Atlassian actually details their architecture (which may have happened? I dunno, I haven't been paying that much attention to it…)

Re: Inside the longest Atlassian outage

#649

Post by [deleted] »

kingofpandora wrote: post #458

Engineering mistakes happen. The most inexcusable thing is not communicating with the paying customers who have been affected for over a week. Atlassian's Global Head of Customer Success probably should have been fired but here she is promoting Atlassian Cloud on LinkedIn three days ago: https://www.linkedin.com/mwlite/in/gertie-rizzo-5b70061 Actually reading a bit more, it seems like their customer team was partying…

[deleted]

Re: Inside the longest Atlassian outage

#650

Post by jupp0r »

Sebguer wrote: post #643
staticautomatic wrote: post #622

Earlier quoted context omitted.

Fair enough, but anyone who’s been around the block on crisis communications knows that you’re supposed to hit the big red “stop all regular communications until we get a fucking handle on this” button.

You think Atlassian should have the ability to stop posts on all of their individual employees' social media posts?

It’s quite common that employees are only allowed to share specific information about companies they work for on social media. Atlassian most likely has that power already.
Post reply on HN