How we built Uber Engineering's highest query-per-second service using Go (2016) • Hacker News Board

Re: How we built Uber Engineering's highest query-per-second service using Go (2016)

#101

Post by dang » Thu, Nov 14, 2019, 7:39 PM UTC

forgottenpass wrote: post #88

thejigisup wrote: post #71

Earlier quoted context omitted.

nah i think its a pretty fair assumption

99% of people building at "scale" are borrowing bragging rights from a large company they happen to be employed by and/or tools they use while doing zero novel work unique to the size of their deployment. To say that these paper tigers are above the bikeshedding of all the plebs is quintessential software echochamber.

"Don't be snarky."

"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something."

https://news.ycombinator.com/newsguidelines.html

We've had to ask you to stop doing precisely this in the past:

https://news.ycombinator.com/item?id=19884138

https://news.ycombinator.com/item?id=13587522

Unfortunately, you've continued to post like this frequently. If you keep doing that we're going to have to ban you, so could you please review the guidelines and fix this?

Re: How we built Uber Engineering's highest query-per-second service using Go (2016)

#102

Post by hobofan » Thu, Nov 14, 2019, 7:55 PM UTC

lclarkmichalek wrote: post #37

sethammons wrote: post #35

Earlier quoted context omitted.

I would argue they should be considering max. Percentiles are good for understanding macro trends. But if every request is sacred, max needs to be monitored. Even at four nines, there are a couple dozen requests per second that are greater than what you are observing. How much greater? Who knows, if you are not observing max. Could be minutes in latency.

Max is pretty horrible, as any comparison requires identical cohort sizes. P99.99 is much better if you want to go wild. Given that the service probably did not have a 100% success rate, and almost certainly had timeouts, the "max" would also likely be at the timeout.

> and almost certainly had timeouts, the "max" would also likely be at the timeout

You filter out the 504s, just the same way you would analyze it on a per-route basis for those metrics.

Re: How we built Uber Engineering's highest query-per-second service using Go (2016)

#103

Post by deminature » Thu, Nov 14, 2019, 7:57 PM UTC

dang wrote: post #95

woeirua wrote: post #36

This is a very inefficient implementation. Really, just poor quality work overall, as anyone with even a basic understanding of spatial indexing would know that an R-tree would be many times faster, as illustrated here: https://medium.com/@buckhx/unwinding-uber-s-most-efficient-s...

This comment is unacceptable on Hacker News, and I'd like to explain in detail why. Please don't think I'm picking on you personally; we all react like this sometimes. Rather, I want to drive this point home to the community. It's important for discussion quality here! ----- Please omit swipes like "Really, just poor quality work" and "anyone with even a basic understanding" from your posts to HN. It's great to add r…

It's disappointing to see a highly-upvoted comment from someone that didn't read the article. The article discusses explicitly why R-trees were not used:

>Instead of indexing the geofences using R-tree or the complicated S2, we chose a simpler route based on the observation that Uber’s business model is city-centric; the business rules and the geofences used to define them are typically associated with a city. This allows us to organize the geofences into a two-level hierarchy where the first level is the city geofences (geofences defining city boundaries), and the second level is the geofences within each city.

Re: How we built Uber Engineering's highest query-per-second service using Go (2016)

#104

Post by spookthesunset » Thu, Nov 14, 2019, 8:03 PM UTC

hobofan wrote: post #102

lclarkmichalek wrote: post #37

Earlier quoted context omitted.

Max is pretty horrible, as any comparison requires identical cohort sizes. P99.99 is much better if you want to go wild. Given that the service probably did not have a 100% success rate, and almost certainly had timeouts, the "max" would also likely be at the timeout.

> and almost certainly had timeouts, the "max" would also likely be at the timeout You filter out the 504s, just the same way you would analyze it on a per-route basis for those metrics.

Sure but now your "100th percentile" would be nothing but the timeout.

Quantifying things like service latency isn't a one-size fits all thing. Every service has its nuances and use cases that make it more meaningful to measure 99%, 99.9%, 99.99% or something else.

.... just don't measure average like I've seen naive junior devs do. Average latency is the worst of all metrics to use as it will include all the outliers at the very tippy top of the spectrum and basically render the metric meaningless.

Re: How we built Uber Engineering's highest query-per-second service using Go (2016)

#105

Post by icholy » Thu, Nov 14, 2019, 8:07 PM UTC

deminature wrote: post #103

dang wrote: post #95

Earlier quoted context omitted.

This comment is unacceptable on Hacker News, and I'd like to explain in detail why. Please don't think I'm picking on you personally; we all react like this sometimes. Rather, I want to drive this point home to the community. It's important for discussion quality here! ----- Please omit swipes like "Really, just poor quality work" and "anyone with even a basic understanding" from your posts to HN. It's great to add r…

It's disappointing to see a highly-upvoted comment from someone that didn't read the article. The article discusses explicitly why R-trees were not used: >Instead of indexing the geofences using R-tree or the complicated S2, we chose a simpler route based on the observation that Uber’s business model is city-centric; the business rules and the geofences used to define them are typically associated with a city. This a…

R-tree will easily outperform that approach.

Re: How we built Uber Engineering's highest query-per-second service using Go (2016)

#106

Post by spookthesunset » Thu, Nov 14, 2019, 8:07 PM UTC

0xEFF wrote: post #55

rayvy wrote: post #44

I think this write up is very fair for a solid engineering team. Is it groundbreaking and eye opening? Absolutely not, I’d say the most “hmm I didn’t know that” part of the entire thing was the part about R-Trees and S2. Is that bad? Absolutely not. These guys did the work, logged their performance and are sharing their story. However, and I believe this is where the animosity in the comments is coming from - given t…

> I guess Leetcoding and Production Engineering really are different things. This is key point. They delivered value to the business. That's the only thing that matters.

> They delivered value to the business.

Sure, but could there have been ways to deliver that value with lower maintenance costs and shorter lead times?

If all you care about is gross revenue and never focus on margins and cost of goods sold then you can justify any project as "adding value to the business".

Re: How we built Uber Engineering's highest query-per-second service using Go (2016)

#107

Post by SilasX » Thu, Nov 14, 2019, 8:12 PM UTC

elbear wrote: post #84

privateprofile wrote: post #66

Earlier quoted context omitted.

> They delivered value to the business. That's the only thing that matters. Except this is an engineering blog post, so the engineering part actually matters. And, as demonstrated in the article [1] linked around this discussion, there were better engineering approaches that would have been even "better for business", as being more efficient means lower costs per transaction and/or higher throughput. [1] https://medi…

It matters for engineers reading the article. It doesn't matter for the business. What they delivered was good enough. Could they have delivered a better solutions? Yes. Should they have searched for a better solution instead of implementing the one they found? They could have spent some time researching, but you can always miss something. It's better to err on the side of delivering something now with a not-so-good…

>It matters for engineers reading the article. It doesn't matter for the business. What they delivered was good enough. Could they have delivered a better solutions? Yes.

The entire reason they posted the article is to brag about having accomplished intelligently. It's relevant if their approach was actually not so intelligent.

"We encountered a standard problem and applied standard solutions" is like "dog bites man". It's not what they were trying to say with the blog post.

>Should they have searched for a better solution instead of implementing the one they found? They could have spent some time researching, but you can always miss something. [...] I'm starting to turn around and focus more on just delivering.

I think the critics point is that the efficient way was probably also cheaper than what they did, and would take the same time to implement, and have lower recurring costs. It would have just been a matter of using off-the-shelf tools and not reinventing the wheel because that wheel is "complicated" and "obviously our case is special". (Someone did benchmarks, and their case is not special.)

You're right, there is a danger to what-if-ing everything and being stuck in decision paralysis. But the clear subtext is that they merit some kind of admiration for how well they did. If that subtext is wrong, it is worth pointing out.

Re: How we built Uber Engineering's highest query-per-second service using Go (2016)

#108

Post by paggle » Thu, Nov 14, 2019, 8:18 PM UTC

woeirua wrote: post #36

This is a very inefficient implementation. Really, just poor quality work overall, as anyone with even a basic understanding of spatial indexing would know that an R-tree would be many times faster, as illustrated here: https://medium.com/@buckhx/unwinding-uber-s-most-efficient-s...

If you follow American football, your comment reads quite a bit like "Tom Brady has terrible throwing mechanics." Ultimately, it might be right, in some purely academic sense, but in a more relevant sense, good throwing mechanics are those that win you games, and good software implementations are those that meet the business requirements.

Re: How we built Uber Engineering's highest query-per-second service using Go (2016)

#109

Post by paggle » Thu, Nov 14, 2019, 8:22 PM UTC

"For each lookup, we first find the desired city with a linear scan of all the city geofences, and then find the containing geofences within that city with another linear scan."

Why wouldn't this be dogshit slow if they used all of the city geofences at once? I would think that first they would scan the country geofences, then the province geofences, then the city geofences, etc...

Re: How we built Uber Engineering's highest query-per-second service using Go (2016)

#110

Post by mbo » Thu, Nov 14, 2019, 9:41 PM UTC

paggle wrote: post #109

"For each lookup, we first find the desired city with a linear scan of all the city geofences, and then find the containing geofences within that city with another linear scan." Why wouldn't this be dogshit slow if they used all of the city geofences at once? I would think that first they would scan the country geofences, then the province geofences, then the city geofences, etc...

It's almost like they could have used some sort of specialised tree data structure where each tree node was assigned to some spatial region on the map, where successive levels were assigned to increasingly smaller regions.