Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Inside the longest Atlassian outage

newsletter.pragmaticengineer.com

Post reply on HN
Print view View original

311–320 of 772 posts

Re: Inside the longest Atlassian outage

#311

Post by hn_go_brrrrr »

baq wrote: post #293
chousuke wrote: post #255

Earlier quoted context omitted.

Beyond trivial scale, you need good processes so that individuals can do their jobs. If you have no processes, change and development becomes extremely difficult because people will be hunting for documentation all the time, stepping on each other's toes, and making mistakes that they should not be making because they forgot a trivial procedure that was a prerequisite to solving their actual problem. I work with a va…

To add to that, and perhaps educate your downvoters a bit, it can be very hard to imagine why or when such strict processes are helpful without having direct experience with organizations of sufficient scale. It literally boggles the mind but the process truly is king when there are hundreds (or thousands) of individuals working on a single product.

Agreed. An essential part of blameless engineering culture is "the outage isn't any one person's fault, it's the fault of the tooling and processes for allowing them to do that". Good processes prevent everyone from making the same mistakes.

Re: Inside the longest Atlassian outage

#312

Post by NationalPark »

kgeist wrote: post #142

We use on-premises setups for almost everything (we generally avoid cloud solutions to have full control of our data), sometimes (approximately once a month) it goes down for a few minutes which already feels like a torture because all our processes depend on it, I can't imagine having no access to it for several weeks, all our work would stop to a halt... The office of the guy who administers on-premise servers is l…

What do you do if the on-premises guy gets hit by a car and isn't in his office?

Re: Inside the longest Atlassian outage

#313

Post by Traster »

I remember finding out one of the senior managers from my company ended up as head of software at Atlassian. It was at that point I was convinced Atlassian has no idea what the hell they're doing. I think this demonstrates the point nicely.

Re: Inside the longest Atlassian outage

#314

Post by kubami »

oauea wrote: post #270
taeric wrote: post #87

Earlier quoted context omitted.

Wait. Why? This sounds like something that feels hard, if you are used to the giant DBs of old. But you can probably get many many instances of the smaller databases without much trouble. Would still be some maintenance, don't get me wrong. But far from impossible.

You'll quickly run into limitations of how many tcp connections you can hold open. Unless you also want to run separate app servers for each customer, which will cost a lot of $$$ Oh, and just forget about allowing your customers to share their data with each other, which most enterprises want in one way or another.

Wait. What? None of the enterprise customers want to share data with each other. And definitely not on a DB level. That should happen in the business logic.

Re: Inside the longest Atlassian outage

#315

Post by jacquesm »

teling2 wrote: post #304
ollien wrote: post #137

Earlier quoted context omitted.

Gotcha. I guess you could be heavy-handed and disable foreign key checks, but who knows what other bugs that would bring into the mix.

The other difficulty is if you don't restore the entire state in a single transaction. Imagine you have partial data restored in Table A but haven't updated Table B correspondingly. Now some other program that consumes Table A and Table B and doesn't have error handling will crash (or worse, mutate state in other weird ways).

That is relational integrity.

Re: Inside the longest Atlassian outage

#316

Post by robertlagrant »

BlargMcLarg wrote: post #211
KptMarchewa wrote: post #193

Earlier quoted context omitted.

One very large video game studio has tons of automation for Jira. Imagine someone deciding to add new weapon. The automation creates 100s of tasks for concept artists, 3d artists, animators, sound artists, software developers with complex dependencies better those. Most importantly, automation creates multiple QA steps for each element of completed work. The same exists for levels, enemies, quests and tons of other e…

See, that is great. Automate what can logically be deduced from the information available and set up templates to provide that information. For developers, it should be automated enough you shouldn't have to write the same info twice, once in commit messages/merges/branch names, once in the ticket itself. If the workflow is so streamlined, all that information can be deduced and the ticket can be advanced automatical…

This is pretty easy with Jira. There's a GitHub plugin which links PRs and commits to a ticket, and a GitHub plugin that links ticket numbers back to Jira tickets.

And you generally do them both at a lower level than tickets, certainly commits, so you don't want to have too much automation between them as that starts adding constraints.

Re: Inside the longest Atlassian outage

#318

Post by ProAm »

brianwawok wrote: post #25
R0ger wrote: post #13

I guess this is wake call for the people rushing to SaaS solutions.

Is it? We use JIRA. Not impacted. If this had hit us.. we would just switch to excel or something for a week/month? But maybe we are a very light user of JIRA. Nothing in there can't be replaced. It's "nice" to be able to go look up a 3 year old bug and which client reported it, but not really crucial for day to day ops.

> We use JIRA. Not impacted.

This time.

Re: Inside the longest Atlassian outage

#319

Post by tyingq »

codycraven wrote: post #185
closeparen wrote: post #136

Earlier quoted context omitted.

JIRA is a framework for making assembly lines out of knowledge workers. When you're a middle manager at a decent sized company, a major problem you face is that the mass of knowledge workers beneath you are opaque : you have no way of knowing whether they're working or not. Another problem you face is that they're uppity : people who went to college and got used to managing their own time now have all kinds of idiosy…

It sounds like you've been hurt by the some terrible management practices, I'm truly sorry that some managers think their job is to control their subordinates. However, regarding ticketing systems, in team environments, it is very effective and helpful to have a system that manages the data about the work that has been completed, is being worked, and is planned to be worked on . Part of that system might be defining…

>However, regarding ticketing systems, in team environments, it is very effective and helpful to have a system

I think the point is that Jira is particularly granular in the way that it lets you do things with permissions, workflow rules, roles, metrics, etc. There's a fair number of places that use that granularity to create a weird digital sweatshop.

Meaning the complaint is more about really deep "micromanagement as a service" than what you might get with lighter tools.

Re: Inside the longest Atlassian outage

#320

Post by pid-1 »

theptip wrote: post #298
closeparen wrote: post #136

Earlier quoted context omitted.

JIRA is a framework for making assembly lines out of knowledge workers. When you're a middle manager at a decent sized company, a major problem you face is that the mass of knowledge workers beneath you are opaque : you have no way of knowing whether they're working or not. Another problem you face is that they're uppity : people who went to college and got used to managing their own time now have all kinds of idiosy…

I think you've got part of the answer here, but are selling it short. Jira is the most complex task-processing rule engine that is also easy enough for a small team to operate, and also has the broadest set of integrated tools of any offering. You can use Jira as a simple Scrum board, a Kanban board, or you can build enforced-process monstrosities. You can build customer-support / internal-helpdesk workflows, or even…

Yeah my team uses Jira to keep track of what we are doing and what we need to do.

Each member can actually organize their sprint and create tasks.

Point assignment is not a big deal, it's just there so we avoid promising more than we can chew.

I've found Jira really pleasant to use for lightweight processes.

Post reply on HN