Octane benchmark retired
v8project.blogspot.com
Octane benchmark retired
1–10 of 17 posts
Re: Octane benchmark retired
#2They probably got tired of FF & Edge beating them in their own benchmark. Can you imagine the conversation with some middle level manager when you were asked to explain that?
Re: Octane benchmark retired
#3They probably got tired of FF & Edge beating them in their own benchmark. Can you imagine the conversation with some middle level manager when you were asked to explain that?
That does seem to be a concern here. They are replacing Octane with what amounts to a-collection-of-stuff. Without a specific workset (ie. a benchmark) there is no clear way to measure one thing against another.
This seems like a mechanism to hide where you stand. Now they can just say "That's not what we find in our tests".
I'm inclined to agree that becnhmarks do become obsolete as workloads change and development targets benchmark pointscoring. The solution is to make a new standardised benchmark, or indeed many of them. If they are clear about what they are measuring then the relative perfomance of different JavaScript engines can be understood by developers to give a nuanced view of what performs well.
Re: Octane benchmark retired
#4Looks like someone at Google realized that Servo (Quantum) is coming
Re: Octane benchmark retired
#5They probably got tired of FF & Edge beating them in their own benchmark. Can you imagine the conversation with some middle level manager when you were asked to explain that?
WebKit is currently king on Octane fwiw.
We still think it's a great benchmark. V8 made the mistake of focusing only on this benchmark, so they didn't know that some of their Octane optimizations were bad for other scenarios. We didn't have that problem because we have always used a broader spectrum of benchmarks.
Re: Octane benchmark retired
#6Looks like someone at Google realized that Servo (Quantum) is coming
Servo uses SpiderMonkey to provide Javascript support, so retiring Octane (a Javascript engine benchmark) would have nothing to do with Servo (a layout engine).
Re: Octane benchmark retired
#7First sunspider, now octane.
This shows that browser vendors are not really good at writing JS benchmarks representative of real world, which is not surprising considering that they are not doing real web development.
Ideally benchmarks would come from developers of popular websites like Facebook, Twitter, etc.
Re: Octane benchmark retired
#8They probably got tired of FF & Edge beating them in their own benchmark. Can you imagine the conversation with some middle level manager when you were asked to explain that?
WebKit is currently king on Octane fwiw. We still think it's a great benchmark. V8 made the mistake of focusing only on this benchmark, so they didn't know that some of their Octane optimizations were bad for other scenarios. We didn't have that problem because we have always used a broader spectrum of benchmarks.
Agreed, Octane was good as far as benchmarks go but you have to balance with a much larger set of benchmarks and always ground your work in real-world scenarios (something we focused heavily on from day one e.g. when we realized having an interpreter really helps startup on lots of real-world apps but doesn't do much for Octane score).
That said, FWIW I think Edge/Chakra gets the crown on Windows!
(I work on Chakra)
Re: Octane benchmark retired
#9Re: Octane benchmark retired
#10They probably got tired of FF & Edge beating them in their own benchmark. Can you imagine the conversation with some middle level manager when you were asked to explain that?
That does seem to be a concern here. They are replacing Octane with what amounts to a-collection-of-stuff. Without a specific workset (ie. a benchmark) there is no clear way to measure one thing against another. This seems like a mechanism to hide where you stand. Now they can just say "That's not what we find in our tests". I'm inclined to agree that becnhmarks do become obsolete as workloads change and development…
We find that http://browserbench.org/Speedometer/ -- which is what v8 is optimizing against now -- is extremely close to real world Discourse (EmberJS, Glimmer) performance.
We always wondered why Android was so crazy ridiculously slow at our real world JS vs. iOS starting in late 2013 onward.. getting worse and worse over 2014 and 2015.. and "gee, we optimized for the wrong thing aka Octane" was a big part of that.
http://benediktmeurer.de/2016/12/16/the-truth-about-traditio...
You can read more about the history at https://meta.discourse.org/t/the-state-of-javascript-on-andr...