Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Inside the longest Atlassian outage

newsletter.pragmaticengineer.com

Post reply on HN
Print view View original

331–340 of 772 posts

Re: Inside the longest Atlassian outage

#331

Post by KronisLV »

grog454 wrote: post #307
a4isms wrote: post #76

Earlier quoted context omitted.

They used to say of Microsoft Word, "Nobody uses more than 5% of its features, but every company uses a different 5%." The saying is apocryphal and unlikely to be accurate, but the shape of the thing its describing applies to almost every piece of enterprise software whether installed on-prem or SaaS. And as another comment points out, at Enterprise scale you can substitute "team" or "group" for customer. Every team…

>"Nobody uses more than 5% of its features, but every company uses a different 5%." >The saying is apocryphal and unlikely to be accurate Well its mathematically impossible to be accurate as soon as you have > 20 users.

> Well its mathematically impossible to be accurate as soon as you have > 20 users.

It's probably in the semantics.

Text input and editing is clearly a part of functionality that's probably used by everyone (or at least most users), so it's not possible for "different 5%" to mean what you're alluding to, maybe the phrasing needs work.

In any given 5% there might be 1-4% of overlap with what others are using and the remainder of that is specific to the company.

Re: Inside the longest Atlassian outage

#332

Post by robertlagrant »

throwaway894345 wrote: post #36
h2odragon wrote: post #4

> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production. Simple concepts aren't that simple at their scale, sure, but I suspect this is skimping details on some truly horrendous monolithic architec…

I would really like to understand who makes the decision to purchase JIRA. It's like the C++ of ticketing software--it does everything because no one wanted to sit down and think critically about the use cases and instead decided it would be easier to say "yes" to every single feature request. It definitely feels like whoever is buying JIRA is not on the team who is using it (maybe IT or finance) because it ticks the…

The reason to buy Jira is that loads of stuff integrates with it, and lots of people know it. Maybe not perfect, but that's why. And unless you're in it all the time, which some people may be, its ergonomics are not as important as, say, an IDE's.

Re: Inside the longest Atlassian outage

#333

Post by bee_rider »

profmonocle wrote: post #274
dxf wrote: post #62

Earlier quoted context omitted.

>Are SLAs even real? SLI: Some metric you use to measure a thing (e.g. uptime, latency, etc.) SLO: Some objective you try to hit, as measured by the SLI (e.g. "99.99% of requests are processed within 3 seconds) SLA: A promise to a customer that they will meet some SLO, and consequences if they don't. If there aren't consequences for not meeting the SLO, then measuring and tracking the metrics is a pointless exercise.…

> and consequences if they don't. And these consequences usually just amount to getting some percentage of your service fees back. I'm sure the affected customers will get their entire monthly Atlassian Cloud fees back. Since this is so severe maybe Atlassian will even give them credits for some # of free months. But there's no way the amount they'll get from Atlassian is going to come close to what they're losing in…

Does that indicate it would be preferable to pay more for a more reliable solution, if such a thing were to exist? Although, it definitely would be hard to quantify 'more reliable' there.

Re: Inside the longest Atlassian outage

#334

Post by kgeist »

NationalPark wrote: post #312
kgeist wrote: post #142

We use on-premises setups for almost everything (we generally avoid cloud solutions to have full control of our data), sometimes (approximately once a month) it goes down for a few minutes which already feels like a torture because all our processes depend on it, I can't imagine having no access to it for several weeks, all our work would stop to a halt... The office of the guy who administers on-premise servers is l…

What do you do if the on-premises guy gets hit by a car and isn't in his office?

There's IIRC 3 or 4 people in their department, they administer the whole building (wifi, security cams, LDAP, etc.), not only the on-premises servers. From what I gathered, our internal systems usually go down due to lack of disk space or some bug in the software which requires merely a reboot, it's not rocket science. Another thing is that our IT department (for internal systems) and the SRE department (for client-facing systems) have 24/7 on-call duty so it's unlikely that no one will respond.

Re: Inside the longest Atlassian outage

#335

Post by Cd00d »

throwaway894345 wrote: post #36
h2odragon wrote: post #4

> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production. Simple concepts aren't that simple at their scale, sure, but I suspect this is skimping details on some truly horrendous monolithic architec…

I would really like to understand who makes the decision to purchase JIRA. It's like the C++ of ticketing software--it does everything because no one wanted to sit down and think critically about the use cases and instead decided it would be easier to say "yes" to every single feature request. It definitely feels like whoever is buying JIRA is not on the team who is using it (maybe IT or finance) because it ticks the…

Interesting take.

Personally, I like JIRA. I think it adds a ton of transparency in our org, and while I've used Trello for personal and home projects, I don't see how it's good enough for business. Trello doesn't even allow for time estimates (last I tried), which for us is part of planning. Search in JIRA is also really good, so no ticket is ever just lost to the ether.

Sure, it's not perfect, and waiting for a board to load is annoying, but for distributed work and visibility, I haven't seen something as professionally useful.

Open to exploring though.

Re: Inside the longest Atlassian outage

#336

Post by jjice »

kikki wrote: post #285
dabeeeenster wrote: post #33

Earlier quoted context omitted.

How else do you run a multitenancy platform?

Not quite the same but at Fandom (Wikia), every wiki has its own DB (over 300,000 wikis), and they are clustered across a bunch of servers (usually balanced by traffic). It works well - but we don't ever really need to query across databases. There's a bunch of logic around instance/db selection but that's about as complex as it gets.

Interesting architecture. From a design point of view, I like the idea of full isolation. From an infrastructure point of view I'm a little scared. I'd assume it's actually not that bad and there's a good way to manage the individual DBs and scale them individually.

Really interested if you can share any details.

Edit: I know each wiki is on a subdomain. Does each wiki also have it's own server?

Re: Inside the longest Atlassian outage

#337

Post by robertlagrant »

sam0x17 wrote: post #213
throwaway894345 wrote: post #36

Earlier quoted context omitted.

I would really like to understand who makes the decision to purchase JIRA. It's like the C++ of ticketing software--it does everything because no one wanted to sit down and think critically about the use cases and instead decided it would be easier to say "yes" to every single feature request. It definitely feels like whoever is buying JIRA is not on the team who is using it (maybe IT or finance) because it ticks the…

And yet, it actually is set up in an extremely opinionated annoying way. For example there is no way to actually assign multiple users to the same ticket, which is a big problem if your org legitimately does pair programming (mine does for juniors)

Having a single owner for each ticket is not a bad idea. You can see contributors in git.

Re: Inside the longest Atlassian outage

#338

Post by 0xbadcafebee »

politelemon wrote: post #31

> it takes between 4 and 5 elapsed days to hand a site back to a customer. Atlassian's SLA page says, Premium Cloud Products 99.9% That's 43 minutes of downtime per month. That works out to, Atlassian can't have any more downtime for the next 14 years. Are SLAs even real? I'm being slightly facetious. From the page text it's just a threshold after which I think you're entitled to some money back for that month.

The typical SLA has no teeth because even if the customer gets their money back, the real harm to the customer may be orders of magnitude greater than what they paid for the service. Some services are contractual or tightly embedded and you know you're not gonna lose the customer if your service goes down frequently. If the service provider doesn't lose money or face, they aren't motivated to prevent the downtime.

One alternative I thought of is the Charity SLA. The service provider pledges to give $5,000 to charity for every minute of downtime. Now everyone within the company knows "if we're down, we're losing thousands of dollars a minute!" and thus will be motivated to ensure the services stay up. But even if the services go down, the company's making tax-free donations, which isn't really bad for anybody. The company could even have a specific downtime goal every year, to make sure their monitoring/alerting/runbooks actually work, and to ensure they donate every year.

Re: Inside the longest Atlassian outage

#339

Post by karaterobot »

throwaway894345 wrote: post #36
h2odragon wrote: post #4

> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production. Simple concepts aren't that simple at their scale, sure, but I suspect this is skimping details on some truly horrendous monolithic architec…

I would really like to understand who makes the decision to purchase JIRA. It's like the C++ of ticketing software--it does everything because no one wanted to sit down and think critically about the use cases and instead decided it would be easier to say "yes" to every single feature request. It definitely feels like whoever is buying JIRA is not on the team who is using it (maybe IT or finance) because it ticks the…

The way it works is, someone always says "Sure, JIRA is bad out of the box, but you can customize it to work the way you want" and there is nobody around to say "so now you have two problems: a bad system that depends on having an expert to make it work the way it should".

Then, you pay for JIRA, and that expert customizes it the way they like. It still doesn't work very well for most people. Nobody likes it except one stakeholder, and the engineering lead who acts as a admin on it. A while later, those people have left the company, and everyone else is out of luck.

Seen this exact scenario play out at two different companies now. Am witnessing it play out in real time at a third.

Re: Inside the longest Atlassian outage

#340

Post by mistrial9 »

ivan_gammel wrote: post #254
Viliam1234 wrote: post #177

Earlier quoted context omitted.

Cheaper than whatever is the open-source alternative?

Free software has zero acquisition cost, but non-zero TCO, which can measure in millions USD (recurring salary of dedicated IT team), depending on the size of organization and complexity of the setup. You will need to maintain on-premise infrastructure, automate backups and recovery, automate security, automate updates (including testing and rollbacks) etc etc, basically doing all the jobs of the people responsible f…

in thirty years of experience, I see this talking point straight from Microsoft anti-Open Source days..

> Free software has zero acquisition cost, but non-zero TCO, which can measure in millions USD

Often a primary driver is exactly the opposite -- for-profit companies are accustomed to paying money for a good or service, with a billing pattern and legal obligations. The company financial deciders do not want a setup that does not have a billing pattern and clear legal obligations. Meanwhile, Open Source Software went from niche to mission-critical in the 2000s via the Internet. For-profit companies (and their publicists) scrambled to explain it, and came up with that exact line repeated again today. I do not blame any person for saying it, it was in print in some reliable place. It does not capture the reality in 2022 IMO.

Post reply on HN