There's a famous 'rebuttal' post to this here https://medium.com/@buckhx/unwinding-uber-s-most-efficient-s...
Thats a great write up. They should hire him on the spot
How we built Uber Engineering's highest query-per-second service using Go (2016)
11–20 of 124 posts
Re: How we built Uber Engineering's highest query-per-second service using Go (2016)
#12> High performance in throughput and latency. In our main data center serving non-China traffic alone, this service handled a peak load of 170k QPS with 40 machines running at 35% CPU usage on NYE 2015. The response time was > Geofence lookups are required on every request from Uber’s mobile apps and must quickly (99th percentile For a 100ms total budget, a ~50ms 99th percentile for a single microservice doesn't soun…
If that's 40 bare metal machines... assuming 60ish HT cores, and 35% CPU... that's not that earth shattering. We do that in Java :)
Re: How we built Uber Engineering's highest query-per-second service using Go (2016)
#13> High performance in throughput and latency. In our main data center serving non-China traffic alone, this service handled a peak load of 170k QPS with 40 machines running at 35% CPU usage on NYE 2015. The response time was > Geofence lookups are required on every request from Uber’s mobile apps and must quickly (99th percentile For a 100ms total budget, a ~50ms 99th percentile for a single microservice doesn't soun…
With that kinda volume the 99th is not interesting at all. Should be looking at the 99.99th.
The latency to human user is what matters. If I get an answer in under 100ms as a user 99% of the time, I don’t care whether 100 or 100K times per day someone else hits that 1% worst case.
Re: How we built Uber Engineering's highest query-per-second service using Go (2016)
#14There's a famous 'rebuttal' post to this here https://medium.com/@buckhx/unwinding-uber-s-most-efficient-s...
well basically ubers blog posts are really low quality and from the outside their engineering descisions are kinda vague. like the switch from postgres to mysql. I'm still clueless how you can have so much money and one of the biggest engineering team, but still can't correctly engineer your stuff. I mean, everybody makes wrong decisions or errors in production code.
From what I gathered off of several HN comments of Uber insiders, the size of their engineering team is actually more likely to be one of the causes for bad engineering decisions. They seem to have too many engineers running around for too little actual work to be done, which results in those engineers coming up with stuff to do to keep them busy (and of course to ensure they appear to be busy and worth their money to their superiors).
One of the best ways to create yourself some work is to not choose an already-proven path to a solution, but invent a new one just for the sake of inventing a new one. Of course that's not how this kind of doing is justified - the justification is usually "the proven path does not scale to our needs" or "by using a special approach adapted to our needs we can be more efficient" or "the proven path is too complex, we can get by with something simpler and easier to maintain". Which might actually all be proper justifications, it's just that you should have some hard proof for these statements, like benchmark results of a comparison of different approaches. That part often gets skipped, which is actually ironic, because doing extensive evaluation and benchmarking and implementing different approaches first before choosing one for production actually serves quite well to create even more work to do.
Re: How we built Uber Engineering's highest query-per-second service using Go (2016)
#15There's a famous 'rebuttal' post to this here https://medium.com/@buckhx/unwinding-uber-s-most-efficient-s...
well basically ubers blog posts are really low quality and from the outside their engineering descisions are kinda vague. like the switch from postgres to mysql. I'm still clueless how you can have so much money and one of the biggest engineering team, but still can't correctly engineer your stuff. I mean, everybody makes wrong decisions or errors in production code.
And none of their blog posts about their own geo stuff ever come close to answer the why. What is wrong with e.g. PostGIS or Elasticsearch? Why do they need to reinvent geo databases several times?
Re: How we built Uber Engineering's highest query-per-second service using Go (2016)
#16There's a famous 'rebuttal' post to this here https://medium.com/@buckhx/unwinding-uber-s-most-efficient-s...
Another rebuttal which is less through due to the benchmark being very synthetic but still points out that Uber's numbers are nothing impressive.
https://www.cybertec-postgresql.com/en/beating-uber-with-a-p...
Re: How we built Uber Engineering's highest query-per-second service using Go (2016)
#17> High performance in throughput and latency. In our main data center serving non-China traffic alone, this service handled a peak load of 170k QPS with 40 machines running at 35% CPU usage on NYE 2015. The response time was > Geofence lookups are required on every request from Uber’s mobile apps and must quickly (99th percentile For a 100ms total budget, a ~50ms 99th percentile for a single microservice doesn't soun…
Per machine, that is 4,250k rps. For any http service we write in Go, that lines up with our general sizing estimates before we start work on a generic new service. After we start to hit the service with load, we start profiling and go from there. Sometimes it gets faster and sometimes it gets slower, but it is usually in the right ball park.
Re: How we built Uber Engineering's highest query-per-second service using Go (2016)
#18Earlier quoted context omitted.
well basically ubers blog posts are really low quality and from the outside their engineering descisions are kinda vague. like the switch from postgres to mysql. I'm still clueless how you can have so much money and one of the biggest engineering team, but still can't correctly engineer your stuff. I mean, everybody makes wrong decisions or errors in production code.
And none of their blog posts about their own geo stuff ever come close to answer the why. What is wrong with e.g. PostGIS or Elasticsearch? Why do they need to reinvent geo databases several times?
Too much time and not enough things to do.
Re: How we built Uber Engineering's highest query-per-second service using Go (2016)
#19> High performance in throughput and latency. In our main data center serving non-China traffic alone, this service handled a peak load of 170k QPS with 40 machines running at 35% CPU usage on NYE 2015. The response time was > Geofence lookups are required on every request from Uber’s mobile apps and must quickly (99th percentile For a 100ms total budget, a ~50ms 99th percentile for a single microservice doesn't soun…
If that's 40 bare metal machines... assuming 60ish HT cores, and 35% CPU... that's not that earth shattering. We do that in Java :)
Don't sell the JVM short. Using modern concurrency models (e.g.: Vertx) it will outperform go in throughput and latency.
Re: How we built Uber Engineering's highest query-per-second service using Go (2016)
#20Earlier quoted context omitted.
And none of their blog posts about their own geo stuff ever come close to answer the why. What is wrong with e.g. PostGIS or Elasticsearch? Why do they need to reinvent geo databases several times?
Too much time and not enough things to do.
How does this happen?! They're so expensive