Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Octane benchmark retired

v8project.blogspot.com

Post reply on HN
Print view View original

11–17 of 17 posts

Re: Octane benchmark retired

#11

Post by om2 »

codinghorror wrote: post #10
Lerc wrote: post #3

Earlier quoted context omitted.

That does seem to be a concern here. They are replacing Octane with what amounts to a-collection-of-stuff. Without a specific workset (ie. a benchmark) there is no clear way to measure one thing against another. This seems like a mechanism to hide where you stand. Now they can just say "That's not what we find in our tests". I'm inclined to agree that becnhmarks do become obsolete as workloads change and development…

We find that http://browserbench.org/Speedometer/ -- which is what v8 is optimizing against now -- is extremely close to real world Discourse (EmberJS, Glimmer) performance. We always wondered why Android was so crazy ridiculously slow at our real world JS vs. iOS starting in late 2013 onward.. getting worse and worse over 2014 and 2015.. and "gee, we optimized for the wrong thing aka Octane" was a big part of that.…

JavaScriptCore (WebKit's JS engine) also optimized for Octane. But we additionally optimized for Speedometer, Sunspider (the not totally broken subset that made it into Jetstream) and JSBench. While Octane isn't the single answer to all JS performance measurement, using it in combination with better things seems better than not using it. To go back to your example, iOS both has way better real-world perf and way better Octane results than Android devices.

BTW benchmarks I'd endorse include MotionMark, Speedometer and Jetstream from http://browserbench.org/ as well as JSBench http://plg.uwaterloo.ca/~dynjs/jsbench/

These are all good at testing different aspects of browser performance including relevant testing of the JS engine.

Re: Octane benchmark retired

#12

Post by ajross »

bozonil wrote: post #7

First sunspider, now octane. This shows that browser vendors are not really good at writing JS benchmarks representative of real world, which is not surprising considering that they are not doing real web development. Ideally benchmarks would come from developers of popular websites like Facebook, Twitter, etc.

That's just a truism about benchmarks. Once all the low hanging fruit is picked they stop showing interesting progress and just become a garden for crazy microoptimizations. The same thing happened with C compilers 10-15 years ago.

Basically, Javascript performance is "done". It's not going to get much faster at this point, and higher performance will come from paradigm shifts (SIMD, WebAssembly, pick your fad).

Re: Octane benchmark retired

#13

Post by cromwellian »

pizlonator wrote: post #5
skrowl wrote: post #2

They probably got tired of FF & Edge beating them in their own benchmark. Can you imagine the conversation with some middle level manager when you were asked to explain that?

WebKit is currently king on Octane fwiw. We still think it's a great benchmark. V8 made the mistake of focusing only on this benchmark, so they didn't know that some of their Octane optimizations were bad for other scenarios. We didn't have that problem because we have always used a broader spectrum of benchmarks.

I don't think they only focused on Octane, it's possible they may have weighted it more highly, I can't speak for the v8 team (I don't work on v8), but that sounds like an oversimplification of what probably happened.

One example could be focusing on throughput instead of startup latency which could yield distorted outcomes for real world apps, or mostly app code that is mostly early-bound style OO code/monomorphic functional code vs code that is hostile to class shape analysis/PIC.

It seems in the last few years, especially with the idea of Web competing with native with stuff like asm.js for games, there was a focus on maximize throughput, but it's rare your typical interactive web app is running code that benefits from that.

Re: Octane benchmark retired

#14

Post by franciscop »

This actually makes sense. When the low level stuff is at a terrible level they made Octane to make that better. Once that was fixed, optimizing for it is counter-productive.

This has a big resemblance IMO to most formal study methods including University and natural languages. Once you know the basics you can optimize for learning or for grades, depending on your goals. They are correlated (but not strongly correlated), but the time investment is not negligible so you'll have to choose either of them.

Re: Octane benchmark retired

#15

Post by vanderZwan »

bterlson wrote: post #8
pizlonator wrote: post #5

Earlier quoted context omitted.

WebKit is currently king on Octane fwiw. We still think it's a great benchmark. V8 made the mistake of focusing only on this benchmark, so they didn't know that some of their Octane optimizations were bad for other scenarios. We didn't have that problem because we have always used a broader spectrum of benchmarks.

Agreed, Octane was good as far as benchmarks go but you have to balance with a much larger set of benchmarks and always ground your work in real-world scenarios (something we focused heavily on from day one e.g. when we realized having an interpreter really helps startup on lots of real-world apps but doesn't do much for Octane score). That said, FWIW I think Edge/Chakra gets the crown on Windows! (I work on Chakra)

Out of curiosity: does only having to target one OS make a difference here? Is it "easier" to get it to run fast? Any worries that you don't have compared to Firefox and Chrome, even though you purely work on the JS side of things?

Re: Octane benchmark retired

#16

Post by pizlonator »

cromwellian wrote: post #13
pizlonator wrote: post #5

Earlier quoted context omitted.

WebKit is currently king on Octane fwiw. We still think it's a great benchmark. V8 made the mistake of focusing only on this benchmark, so they didn't know that some of their Octane optimizations were bad for other scenarios. We didn't have that problem because we have always used a broader spectrum of benchmarks.

I don't think they only focused on Octane, it's possible they may have weighted it more highly, I can't speak for the v8 team (I don't work on v8), but that sounds like an oversimplification of what probably happened. One example could be focusing on throughput instead of startup latency which could yield distorted outcomes for real world apps, or mostly app code that is mostly early-bound style OO code/monomorphic f…

Octane was V8's primary tuning benchmark. They tuned for it to the detriment of other benchmarks. Whether you call that "only focusing" on Octane or not is a matter of semantics to me.

Re: Octane benchmark retired

#17

Post by bterlson »

vanderZwan wrote: post #15
bterlson wrote: post #8

Earlier quoted context omitted.

Agreed, Octane was good as far as benchmarks go but you have to balance with a much larger set of benchmarks and always ground your work in real-world scenarios (something we focused heavily on from day one e.g. when we realized having an interpreter really helps startup on lots of real-world apps but doesn't do much for Octane score). That said, FWIW I think Edge/Chakra gets the crown on Windows! (I work on Chakra)

Out of curiosity: does only having to target one OS make a difference here? Is it "easier" to get it to run fast? Any worries that you don't have compared to Firefox and Chrome, even though you purely work on the JS side of things?

I'm sorry I missed your comment earlier, but hopefully replying later is better than never :)

ChakraCore is cross-platform now so we no longer get the benefit of a single platform. However back when we were only Windows there were advantages, certainly, if only because all the work spent getting everything working on different OSes can instead be invested in improving performance.

Post reply on HN