Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Alice is impatient

brooker.co.za

Post reply on HN
Print view View original

1–10 of 46 posts

Re: Alice is impatient

#2

Post by trb »

Considering other metrics then p99 for user impact is unwise. All users will at some point experience a By focusing on the tail and optimizing worst cases you help users more than by improving your median latency.

Re: Alice is impatient

#6

Post by perching_aix »

I've grown to dislike the typical tail measurements completely. What I usually look at these days is what share of unique users experience an "unacceptable experience" over a measurement period instead.

I find it much more inquisitive and visceral, to the extent that p99 now boggles my mind. 2N would be dreadful as an availability figure, yet for UX it's treated very different. So much so that my measurements corroborate exactly that; good UX requires the same many-nines reliability as e.g. DCs, not one or two.

I wonder if it's p90 and p99 to blame for the shoddy services we have, in a way. It's pretty hard to argue for fixing something when it's presented as only going wrong 0.5% or less of the time after all. Even if at scale that means most of your users are experiencing it weekly.

Re: Alice is impatient

#7

Post by AgentOrange1234 »

rustybolt wrote: post #4

This article contains very little substance. Show me the math!

Yes I found this very hard to follow. I appreciate expressing ideas in math like E_a[X] as much as the next guy, but there is no definition or even description of what the heck E or E_a or Var(x) even mean, so how is anyone supposed to understand the reasoning here? All I get from this is a claim that experienced latency is different than the mean, which sounds important, but I still have no intuition as to why this is. Which is sad, because Booker's blog is often deeply amazing.

Re: Alice is impatient

#8

Post by NightMKoder »

AgentOrange1234 wrote: post #7
rustybolt wrote: post #4

This article contains very little substance. Show me the math!

Yes I found this very hard to follow. I appreciate expressing ideas in math like E_a[X] as much as the next guy, but there is no definition or even description of what the heck E or E_a or Var(x) even mean, so how is anyone supposed to understand the reasoning here? All I get from this is a claim that experienced latency is different than the mean, which sounds important, but I still have no intuition as to why this…

This is standard statistics terminology - E(X) is https://en.wikipedia.org/wiki/Expected_value . E_a is presumably Alice's perceived expected value. Var(X) is https://en.wikipedia.org/wiki/Variance . The law of large numbers says the arithmetic average of observations becomes E(X) with enough samples.

I'm pretty sure what the author is saying is:

E(X) =:= \sum_t(t * P(X = t)) is the definition

another important note is P(X^2 = t^2) = P(X = t) - because it's the same distribution.

E_a(X) is a bit sloppy, but consider X_a aka Alice's latency "experience" distribution. The argument is:

P(X_a = t) = t * P(X = t) / \sum_u(u * P(X = u)) - i.e. scale the probability up by t but make it sum to 1.

Then

E(X_a) = \sum_t(t * P(X_a = t)) = \sum_t(t * t * P(X = t) / \sum_u(u * P(X = u))

aka

E(X^2) / E(X)

Then (from wikipedia)

Var(X) = E(X^2) - (E(X))^2

And we get

E(X_a) = (Var(X) + (E(X))^2) / E(X) = E(X) + Var(X) / E(X)

Re: Alice is impatient

#9

Post by kgwgk »

zaik wrote: post #5

Is the formula for E_a[X] trivial? I don't see it immediately...

E[X^2] weights each time with the time, giving the square, and the E[X] in the denominator is the normalisation factor (also required to fix the dimensions).

Say that there are to different waiting times 1s and 3s, and they happen with probability 50% each. The average waiting time (1/2 1+1/2 3) is 2s. However, 75% of the time we are waiting on a 3s event and only 25% on a 1s event. The weighted average is 2.5s. E[X^2]=1/2 1+1/2 9=5(s^2) is not the right answer, it still has to be divided by E[X]=2(s) to get the correct answer.

Re: Alice is impatient

#10

Post by ggm »

Interesting you work at Amazon and show how end user experience weights to their pessimal experience.

So.. apply that to Amazon design heuristics like author name search on books, and how Amazon return "in the style of" and "not a book but this guy called Charles Dickens makes jigsaws" as high order matches and consider how the end user experience weights to the pessimal yet Amazon can show on average they make more money doing this..

(Understood that engineers and AWS don't influence UX in the storefront or search)

Post reply on HN