Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we built Uber Engineering's highest query-per-second service using Go (2016)

eng.uber.com

Post reply on HN
Print view View original

111–120 of 124 posts

Re: How we built Uber Engineering's highest query-per-second service using Go (2016)

#111

Post by xtomus »

icholy wrote: post #105
deminature wrote: post #103

Earlier quoted context omitted.

It's disappointing to see a highly-upvoted comment from someone that didn't read the article. The article discusses explicitly why R-trees were not used: >Instead of indexing the geofences using R-tree or the complicated S2, we chose a simpler route based on the observation that Uber’s business model is city-centric; the business rules and the geofences used to define them are typically associated with a city. This a…

R-tree will easily outperform that approach.

You can either ship a 2 dimensional linear search approach today or ship an R tree based approach in 2 weeks time. I know which one in choosing if they both meet all of the requirements

Re: How we built Uber Engineering's highest query-per-second service using Go (2016)

#112

Post by woeirua »

deminature wrote: post #103
dang wrote: post #95

Earlier quoted context omitted.

This comment is unacceptable on Hacker News, and I'd like to explain in detail why. Please don't think I'm picking on you personally; we all react like this sometimes. Rather, I want to drive this point home to the community. It's important for discussion quality here! ----- Please omit swipes like "Really, just poor quality work" and "anyone with even a basic understanding" from your posts to HN. It's great to add r…

It's disappointing to see a highly-upvoted comment from someone that didn't read the article. The article discusses explicitly why R-trees were not used: >Instead of indexing the geofences using R-tree or the complicated S2, we chose a simpler route based on the observation that Uber’s business model is city-centric; the business rules and the geofences used to define them are typically associated with a city. This a…

I did read the article before posting. They don’t actually explain why they opted to not use an R-tree except to say that they think this organizational structure makes sense intuitively. Unfortunately, intuition here is wrong, and the link I posted specifically benchmarks their solution and shows that if they had done even basic benchmarking they would have seen this.

Now there might be a valid argument for this approach if the geofences are changing constantly and reindexing the R-trees would take too long, but in the end they synchronize everything anyways, and the R-tree could easily be generated on another node, serialized and then unserialized asynchronously before swapping to an updated tree.

Re: How we built Uber Engineering's highest query-per-second service using Go (2016)

#113

Post by woeirua »

xtomus wrote: post #111
icholy wrote: post #105

Earlier quoted context omitted.

R-tree will easily outperform that approach.

You can either ship a 2 dimensional linear search approach today or ship an R tree based approach in 2 weeks time. I know which one in choosing if they both meet all of the requirements

Just about every modern language has a well established R-tree library...

Re: How we built Uber Engineering's highest query-per-second service using Go (2016)

#114

Post by bborud »

asdfman123 wrote: post #28
sidlls wrote: post #27

Earlier quoted context omitted.

They also would have to sift through some really terrible programmers who have fully bought into the worst practices of enterprise-y OO development.

Is that really true? Honest question. Or is that just bias against things that are not new and fancy?

In my experience: yes, this is true. I’m not going to speculate about why, but I suspect this will eventually happen to Go as well when/if it becomes a more mainstream language.

Re: How we built Uber Engineering's highest query-per-second service using Go (2016)

#115

Post by DuskStar »

stefano wrote: post #97
CamouflagedKiwi wrote: post #90

Earlier quoted context omitted.

There are dramatic memory and startup time improvements. Much less so for CPU once everything is up and running.

Is that still true after tuning the JVM GC for a low memory footprint? I'm genuinely asking, I'd like to see an article on the subject. I've read in the past that the default settings are geared towards long-running processes and trade off memory usage for higher throughput, as in general there's no free lunch with GCs. Better memory usage always implies worse performance (and viceversa), just like a higher throughpu…

There's two more variables you're missing here, besides "memory" and "pause times" - how well matched to GC the language is, and how advanced/good the GC is. Some languages might just be fundamentally easier to garbage collect than others - a simple example might be that of a language with no references and no threading, where the GC is going to be extremely simple. And GC quality is certainly a thing, since I doubt major companies would spend tens/hundreds of millions of dollars producing GC 'advancements' that just move a slider back and forth.

Re: How we built Uber Engineering's highest query-per-second service using Go (2016)

#116

Post by apta »

dashwav wrote: post #30
matt2000 wrote: post #23

Earlier quoted context omitted.

My sense is that most people moving to Go are coming from Node and that explains their excitement about the performance. Not many upsides when you’re already on Java.

I mean there is a definite performance gain from Java to Go, although I would agree most people probably wouldn't hit it - but if its just as easy to write, might as well use the better language. On top of that Go makes concurrency so much easier to factor in and program with, which in the backend server space is a very important asset over Java. As an interesting addendum, I found this to be a really interesting res…

> might as well use the better language

Which would be Java in this case.

Re: How we built Uber Engineering's highest query-per-second service using Go (2016)

#117

Post by icholy »

xtomus wrote: post #111
icholy wrote: post #105

Earlier quoted context omitted.

R-tree will easily outperform that approach.

You can either ship a 2 dimensional linear search approach today or ship an R tree based approach in 2 weeks time. I know which one in choosing if they both meet all of the requirements

I mean why stop there? Let's remove the database indexes too. Hell, why use a hashtable when you have lists amirite?

Re: How we built Uber Engineering's highest query-per-second service using Go (2016)

#118

Post by sgt »

YawningAngel wrote: post #26
sgt wrote: post #25

Earlier quoted context omitted.

I've changed my opinion a bit about Vertx combined with Java. Personally I tried to push it for several years, but most Java developers (and I mean 95% in my particular case) seem to start resenting it over time. It's hard to write good services in Vertx, mostly due to its asynchronous model combined with Java's verbosity and boiler plate. Many teams have junior developers and IMHO it's simply not safe expecting them…

I suspect that Loom Fibers are going to offer parity with golang in this area soon enough.

Loom seems very interesting. Do you know how far off they are currently?

Re: How we built Uber Engineering's highest query-per-second service using Go (2016)

#119

Post by overcast »

asdfman123 wrote: post #98
overcast wrote: post #62

Earlier quoted context omitted.

Why is it that every Java application I've encountered in my 20 years of corporate IT, has been a performance pig? Is it just a case of bad developers, making shit code? Is it difficult to make Java perform well, but when it does, it shines? I can't have accidentally interacted with ONLY the crappiest Java apps in my tenure.

It's possible to make bloated crap in any language. But enterprise programmers making internal software have little incentive to make their programs sleek and fast because they 1) have a captive user base who are forced to use their bad UIs, and 2) there aren't many users so you don't really need to optimize for minimal server use. From the eyes of management, a good enterprise programmer can take a request from star…

This isn't even internal stuff, all paid commercial applications. Oracle is notorious for it, but most recently dealing with SysAid(Helpdesk software).

Re: How we built Uber Engineering's highest query-per-second service using Go (2016)

#120

Post by CamouflagedKiwi »

stefano wrote: post #97
CamouflagedKiwi wrote: post #90

Earlier quoted context omitted.

There are dramatic memory and startup time improvements. Much less so for CPU once everything is up and running.

Is that still true after tuning the JVM GC for a low memory footprint? I'm genuinely asking, I'd like to see an article on the subject. I've read in the past that the default settings are geared towards long-running processes and trade off memory usage for higher throughput, as in general there's no free lunch with GCs. Better memory usage always implies worse performance (and viceversa), just like a higher throughpu…

Yes.

Every object in Java has something like three words of overhead; also since it doesn't have value semantics (yet) objects are typically allocated out-of-line, so an ArrayList makes a linear number of allocations, whereas a Go slice makes a constant number. Plus the binary sizes are typically much smaller; not really an expert but our observataion at work is non-trivial Java services allocate a lot of memory through classloading / JITing whereas a Go binary will typically be very small.

Basically, agreed on the GC tradeoff, but the higher memory footprint of Java mostly comes from other areas.

> Better memory usage always implies worse performance

That isn't really true. Better memory usage implies better cache-friendliness (and possibly better locality too).

Post reply on HN