371–380 of 772 posts
Earlier quoted context omitted.
Now every time you run a database migration, you have to adjust N tables - and in Atlassian's case, N is 200000. Is that better? It depends. There is no "best" way of doing multitenancy.
That’s just an automation issue. It’s not like you have to write a bespoke database migration script per DB.
Earlier quoted context omitted.
I can't believe that they would intermix the data in that way... but if they did, godspeed to them, they're likely still overpromising what can be done in this time frame.
They don't, you're responding to speculation which is just outright wrong. Jira and Confluence is single tenanted databases, unless something fundamental has changed at Atlassian in the past 4 years. Source: worked at Atlassian, on Jira, 4 years ago.
> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production. Simple concepts aren't that simple at their scale, sure, but I suspect this is skimping details on some truly horrendous monolithic architec…
Yeah, I’m thinking the exact same thing. Perhaps they don’t have the right people on hand to do hard things like this. They also apparently lack an incident response plan since a critical component of that is coms to affected customers. They also lack good practices around preventing human error. It should not have even been possible to make the initial mistake. It certainly should have involved multiple steps of “ar…
I am curious if anyone can provide any more insight on this simplification.
I've worked at companies like this. Originally a core of motivated creative individuals make a cool product. As the business grows rapidly, Pournelle's (Iron) Law (of Bureaucracy) takes over. For a variety of reasons, the very capable creators depart and are replaced by less motivated/aware individuals who are glad to have a job and easily compelled to do things to the product that probably should not be done.
My guess is that while Atlassian may have originally been one of those cool founder places, it has probably morphed into the more incompetent version that comes with scale all too often. But I don't know. Thus my question if anyone can speak to the true current tech capabilities of this company.
Earlier quoted context omitted.
Not in a negative way. You want to trust engineers to always have changes built and tested before they go to production, but when something egregious happens you need to go back and see what went wrong. You can choose to interpret that as control, but really the only alternative (often cited) is "Well that shouldn't ever happen, so you don't need tooling to support that situation". And that is not a useful way of thi…
I think the problem is that the processes are often not mutually agreed , but instead dictated by middle managers. JIRA then becomes a tool for enforcing arbitrary rules, e.g. control
Earlier quoted context omitted.
I would really like to understand who makes the decision to purchase JIRA. It's like the C++ of ticketing software--it does everything because no one wanted to sit down and think critically about the use cases and instead decided it would be easier to say "yes" to every single feature request. It definitely feels like whoever is buying JIRA is not on the team who is using it (maybe IT or finance) because it ticks the…
JIRA is a framework for making assembly lines out of knowledge workers. When you're a middle manager at a decent sized company, a major problem you face is that the mass of knowledge workers beneath you are opaque : you have no way of knowing whether they're working or not. Another problem you face is that they're uppity : people who went to college and got used to managing their own time now have all kinds of idiosy…
One JIRA is the project that's used for development of the core product, where there are no constraints— anyone can add a comment, create links, change assignee, add new tags, push the tickets through whatever state transitions they want, and so on. It works, though it is a little chaotic sometimes as subgroups of people have different preferences for how things should go (eg, for tickets requiring test team validation, should the ticket assignee remain as the person who did the original work so it's clear who has more to do if it fails validation, or should the assignee change to the test team person, so that it's clear that that's the next person who has it as an action item?)
The second JIRA is the IT team's internal support project, which is completely locked down— no one except them can close tickets or move them around, or even edit the contents, closed tickets can't be commented on any more, and so on. This is the one that gives me the vibes you are talking about. Every time I have to interact with it, I loathe it because every inch of it is transparently a funnel, railroading me along a path toward one of either DONE or WONTFIX. This is absolutely efficient, in the sense of meeting the goal of closing all the tickets, but I feel it introduces friction for the larger business goal of actually helping people resolve their problems. To the point where eventually most of the IT support activity moved away from the JIRA project to an informal Slack channel, which is way more accessible, but worse in basically every other way: it's harder to effectively search, impossible to properly link, bad for async, bad for dealing with more than one thing at once, etc.
Earlier quoted context omitted.
That’s just an automation issue. It’s not like you have to write a bespoke database migration script per DB.
The bug we are mitigating was also just an automation issue.
Automation wasn’t the issue here. It’s the symptom not the cause.
> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production. Simple concepts aren't that simple at their scale, sure, but I suspect this is skimping details on some truly horrendous monolithic architec…
Personally, given the multi-day outage, I think I would just restore everything to a separate system, and then only point the affected customers to this new system. Take the hit of having two fully separated system initially, and work on reconciling them after without having to worry about the on-going outage. I wonder if they're not doing this due to some tech limitations, to avoid taking the financial cost of runni…
For just this sort of problem, we actually had three DB servers running all the time: active, passive, and hour behind with the ability to break hour behind's copying of the write-ahead log of active as the DBA's secret weapon for just this problem. If all customers had accidentally lost an hours worth of data it would have been embarrassing, but much less than completely shutting out hundreds of paying customers for two weeks, I think?