Atlassian Cloud ToS section 3.3(I) prohibits discussing performance issues • Hacker News Board

Re: Atlassian Cloud ToS section 3.3(I) prohibits discussing performance issues

#201

Post by acdha » Sat, Jan 02, 2021, 11:30 PM UTC

confluence_perf wrote: post #199

meibo wrote: post #190

Earlier quoted context omitted.

This is even worse than the crazy slowness. I spend 10 minutes to make a detailed bug report, just to have it fall apart after submitting. How does that happen in a software made to show bug reports? Just use standard markdown instead of your own bullcrap formatting that doesn't ever seem to work.

Best I can say here (as a Confluence PM not a Jira one, and in a public forum) is that customers are not the only ones that experience this pain, and the appropriate folks are notified on a regular basis. I think the genesis of this is historically many different editable fields might had different modules behind them (not every single one a different one, but maybe a handful of common ones). It looks like for whatev…

I’d bet on legacy code which nobody wants to touch and we’ve all been there. I’d recommend having an option to add a Markdown flag which can be enabled for all new records, leaving the old stuff alone, since this usually comes up in the context of people using GitLab instead because it’s so, so much better for technical work. Once Jira is just links to Gitlab, people start asking what justifies the hefty cost for a tool the team doesn’t use.

Re: Atlassian Cloud ToS section 3.3(I) prohibits discussing performance issues

#202

Post by ineedasername » Sat, Jan 02, 2021, 11:34 PM UTC

jiggawatts wrote: post #125

sytse wrote: post #44

Earlier quoted context omitted.

Those suggestions in your last paragraph look very reasonable to me. At GitLab we explicitly allow performance testing as our ninth and final stewardship promise https://about.gitlab.com/company/stewardship/#promises But I recognize there is a trade-off and companies can reasonably but the balance at different points.

There is no tradeoff. DeWitt clauses are corporate censorship and are 100% self-serving. There is zero benefit to consumers having no benchmarks at all available for entire product categories. There is an enormous benefit to corporations to be able to silence critics with the threat of bankruptcy via lawsuit. This is big corporations using the law to bully journalists and citizens, nothing more.

Pretty much, yes. That's why I included an example of how they could allow transparent assessment while still addressing their stated concerns. Because it shows the stated concerns, in having a solution, are just a BS smoke screen. (Not that my proposed scenario is still the best case for consumers, just that there's a way they could allow public benchmarks without sacrificing their concerns)

Re: Atlassian Cloud ToS section 3.3(I) prohibits discussing performance issues

#203

Post by acdha » Sat, Jan 02, 2021, 11:35 PM UTC

confluence_perf wrote: post #98

acdha wrote: post #89

Earlier quoted context omitted.

> We have certainly heard from some customers that agree that 'everything' is slow, but we've also heard from other customers saying they have no problems. What do your metrics show? I instrument my web sites so I know how long every operation – server responses, front-end JS changes, etc. – takes and can guide my development accordingly. You have a much larger budget and could be answering this question with hard da…

Hi acdha, We have metrics, but of course as with many such things you always want more insights than the amount of data you're collecting (so we're always trying to grow this as appropriate). This data is what led to the above (added edit trying to reply to @rusticpenn) saying we can see that "some instances are slower than others", and "some users are slower than others". I can't share those numbers of course though…

I mean, it really is everything. If it were my project I’d make sure I have telemetry on all UI actions and would then set a threshold (say 200ms), triage everything which has a percentile consistently over that threshold to look for easy fixes, and then set a policy that each release only improves on those numbers. I can’t think of any user-visible changes to Jira or Confluence in the last 5 years which I wouldn’t trade in a heartbeat for good performance.

Re: Atlassian Cloud ToS section 3.3(I) prohibits discussing performance issues

#204

Post by BillinghamJ » Sat, Jan 02, 2021, 11:36 PM UTC

confluence_perf wrote: post #195

BillinghamJ wrote: post #161

Earlier quoted context omitted.

I'm afraid I'm no expert on project management tools! On what users experience as effectively "instantaneous", that's from experience on UX engineering and industry standards - https://www.nngroup.com/articles/response-times-3-important-... On the other noted times, they're just a general range of what can be expected from a reasonably well-built tool of this nature. Obviously much simpler systems should be drastical…

Thanks for the link! Yes this is the general guidance we're using too (0.1/1/10s), and one that we're reinforcing at every level of the company. This link does have more detail than I've seen in other places though, so it's an interesting read. However I've not seen guidance on whether these should be P90 or P95 or P99 measures for example though. We've selected something internally, but obviously selecting amongst t…

The percentiles are a bit of a combination.

A big part is simply how far you are in your journey of getting good at performance - if your p50 is still garbage, there's not much point in focussing on your p99 measurements. You should be targeting the p99 long term, but focus on the p50/p90 for now.

It's super important to target and make long term decisions around the p99 though, because, e.g., making a 100x improvement is not possible through little iterative changes over 2-3 years. You need a base to work from where that 100x is fundamentally achievable, which requires thinking from first principles and slightly getting out of the typical product mindset.

I also find the typical product mindset tends to result in focussing a lot on the "this quarter/next quarter" goals, but neglecting the "8/12 quarters from now" as a result.

Beyond short term/long term goals, the choice is largely just down to what the product is/does. Even ignoring all current architectural choices, there are some fundamentals where certain things must always be faster/slower - e.g. sync writes will typically be a fair bit slower than reads, and typically occur much less often, complex dynamic queries which can't be pre-optimised require DB scanning but are much less common.

For these kinds of tools, where most of the interaction is reads, mostly on predefined or predefined + small extra filtering, and reading/writing on individual resources (ie tickets), you can get p99 numbers trending towards the 100ms mark eventually - there's very little which truly can't get to that level with clever enough engineering.

---

Of course I imagine Google tends to be looking more at their p99.9/p99.99/pmax/etc(!), at least for their absolute highest volume systems.

None of us are going to be getting to that point, but it's often worth thinking about engineering principles against a super high bar - it often helps people to open their minds a bit more and think more outside the box when given a really dramatic goal magnitudes beyond their existing mindset.

Of course you're not expecting to really get to that level, but anchoring that way can achieve amazing things. I've done that with a lot of success at my company and we actually did manage to achieve a few originally thought to be totally unrealistic.

Re: Atlassian Cloud ToS section 3.3(I) prohibits discussing performance issues

#205

Post by deathweasel » Sat, Jan 02, 2021, 11:39 PM UTC

confluence_perf wrote: post #153

BillinghamJ wrote: post #150

Earlier quoted context omitted.

As a first step, 1s would be better than nothing for sure, but you need to be working towards a much tighter goal on a 1-2 year timeframe. New load, you should really be hitting 200ms as your 95th percentile - 300ms or so would be decent still. "Transitions" should hit 100ms 95th, 150ms would be decent. If you did hit 100ms across the board, you'd be rewarded by your customers/users psychologically considering the in…

Thank you for coming back and clarifying. Do you happen to have links to any public testing results of other tools, or guidance to this specificity - would love to use them to build a case internally Most of what we've seen online are nowhere near this level of detail (X-ms for Y-%ile for Z-type of load) (edit: clarified request)

Do you guys use synthetic monitoring tools?

Re: Atlassian Cloud ToS section 3.3(I) prohibits discussing performance issues

#206

Post by confluence_perf » Sat, Jan 02, 2021, 11:42 PM UTC

acdha wrote: post #203

confluence_perf wrote: post #98

Earlier quoted context omitted.

Hi acdha, We have metrics, but of course as with many such things you always want more insights than the amount of data you're collecting (so we're always trying to grow this as appropriate). This data is what led to the above (added edit trying to reply to @rusticpenn) saying we can see that "some instances are slower than others", and "some users are slower than others". I can't share those numbers of course though…

I mean, it really is everything. If it were my project I’d make sure I have telemetry on all UI actions and would then set a threshold (say 200ms), triage everything which has a percentile consistently over that threshold to look for easy fixes, and then set a policy that each release only improves on those numbers. I can’t think of any user-visible changes to Jira or Confluence in the last 5 years which I wouldn’t t…

Hi acdha,

Thank you for the advice - how about on the page load side? Our biggest problem is probably the variance issue (mentioned in other subthreads) -> we can't easily tell what is the difference between a slow and a fast load in many cases.

Even if we compare things that are available from the metrics like CPU, Mem, and Network speed, those are not very granular metrics (for example, to understand someone with 16 threads was actually at 95% Mem used during that page load), and have little correlation at a wide level with page load speed.

Re: Atlassian Cloud ToS section 3.3(I) prohibits discussing performance issues

#207

Post by paulryanrogers » Sat, Jan 02, 2021, 11:42 PM UTC

cortesoft wrote: post #24

paulryanrogers wrote: post #11

The most innocent explanation is preventing benchmarks against similar tools which could be unflattering. Though the only other company that resorts to such insecure measures is Oracle.

The comment right next to yours is a link to cloudflare's TOS with the exact same provision. It is not just Atlassian and Oracle.

I meant to include "that I know of" but was too lazy to edit. Anyway, it's still a bad sign IMO. Signaling desperation instead of concern for user experience.

Re: Atlassian Cloud ToS section 3.3(I) prohibits discussing performance issues

#208

Post by jjcm » Sat, Jan 02, 2021, 11:47 PM UTC

random3 wrote: post #20

If you start using Linear you won't need a benchmark to notice the difference. This is what happens when the "clueless" start "innovating". I've had several conversations over the years with members of Atlassian technical teams. They always wanted to work on performance, but never allowed (priorities). They are in "good" company (Oracle, China, etc.). What's preventing anonymous performance benchmarks, though?

> I've had several conversations over the years with members of Atlassian technical teams. They always wanted to work on performance, but never allowed (priorities).

For what it's worth, it's a pretty significant company priority now. I recently had my project (dark mode) cancelled[1] so we could dedicate engineering effort to performance.

[1] https://jira.atlassian.com/browse/JRACLOUD-63150?focusedComm...

Re: Atlassian Cloud ToS section 3.3(I) prohibits discussing performance issues

#209

Post by confluence_perf » Sat, Jan 02, 2021, 11:52 PM UTC

deathweasel wrote: post #205

confluence_perf wrote: post #153

Earlier quoted context omitted.

Thank you for coming back and clarifying. Do you happen to have links to any public testing results of other tools, or guidance to this specificity - would love to use them to build a case internally Most of what we've seen online are nowhere near this level of detail (X-ms for Y-%ile for Z-type of load) (edit: clarified request)

Do you guys use synthetic monitoring tools?

I don't think I know what that might exactly mean - I know we have synthetic traffic generation tools (and thus, measurements generated from the synthetic traffic), but I think those exhibit the same variance as production -> the backend for them are the same cloud IaaS systems and SW, so there's no 'sandboxed from all outside variance'.

If it means something else then I'm not aware if we do it or not.

Re: Atlassian Cloud ToS section 3.3(I) prohibits discussing performance issues

#210

Post by acdha » Sat, Jan 02, 2021, 11:53 PM UTC

smarx007 wrote: post #134

dylan604 wrote: post #80

Earlier quoted context omitted.

If you need tips or TRICKS to make a product useable, you have a BAD product.

Well, you certainly need "tips and tricks" to make Arch Linux fully usable. Every powerful tool needs to be adjusted to its use (Github is full of dotfiles and macOS bootstrap repos). Doing so is a sign of professionalism (craftsmanship).

I think that’s saying more about Arch than anything else. While it’s not wrong that people add dot files to track preferences, those aren’t necessary for basic usage. Someone using macOS without customizing anything still has a good experience.

A key distinction is between domain complexity and what the product adds to that intrinsic complexity. If you use GitHub or GitLab issues with no customization you’ll have a better experience than a Jira user because they work well out of the box without requiring customization or adjusting your workflow just to accomplish core tasks.