Earlier quoted context omitted.
Doesn't the Netflix test really test GPU decoding vs software decoding? I'm not 100% sure on this, so any clarification welcome, but I thought that Netflix content contracts DRM requirements allowed GPU decoding via Playready, but not on some others like Widevine. Would the Netflix test still show such an advantage for Edge if there weren't any DRM?
It seems to me that if you're running on the same device, the gpu should not be a variable.
Why we challenge Microsoft’s battery test
131–140 of 220 posts
Re: Why we challenge Microsoft’s battery test
#132Wow thanks for the moderation! I guess here in Hacker News we are all smart engineers and scientists that rely on the scientific method except for when it comes to microsoft when we all turn into drooling idol worshiping children. I wonder whether MS has a well orchestrated campaign to have multiple employees troll hn, or whether they just go to the source and buy moderation points in bulk from ycombinator.
People post things on HN all the time that criticize Microsoft without getting downvoted — because they do it without antagonizing other HN users.
Re: Why we challenge Microsoft’s battery test
#133Edge platform team employee here. A couple thoughts on methodology, focusing on our scripted test (under "stay productive longer" at https://blogs.windows.com/windowsexperience/2016/06/20/more-... ), which is most comparable to Opera's test; nobody seems to challenge our Netflix results: * We did not enable ad blockers, because we were testing the efficiency of the network stack, browser rendering pipeline, etc.; if…
How did you control for the specific ads the browsers were served? Ads are highly dynamic. Unless you (can) include the differences in your methodology you better eliminate that influence.
Re: Why we challenge Microsoft’s battery test
#134Earlier quoted context omitted.
How did you control for the specific ads the browsers were served? Ads are highly dynamic. Unless you (can) include the differences in your methodology you better eliminate that influence.
Ding, ding, ding and to be fair this is the problem in both methodologies. The only way to accurately perform this test is to spider a bunch of sites, save the contents to a locally hosted HTTPD and ensure all third-party JS calls are resolved locally and test both against the exact same sites as they were spidered at a point in time. You can simply not account for changes which may happen to the markup, ad network b…
I don't see anything that would somehow create a bias in favor of a specific browser as far as ad networks goes, if anything the stigma/stereotyping of IE/Edge users would probably mean that ad networks are more incentivized of sending the baity apps towards those browsers.
As for the network part well again that's an important metric if certain browsers perform better at adverse network conditions it's an important factor to know, you do not want to give them the best case scenario every time.
Giving a page a fixed amount of seconds to load is also completely the wrong approach you want to see how browsers behave when they can't load a page properly or when it takes more time than usual, maybe some browsers expend more resources by resubmitting the entire request, maybe some browsers do not parse the DOM tree from scratch when some of the requests stall, maybe some browsers have less resource intensive placeholders for DOM elements, maybe some browsers are better at adjusting the DOM preprocessor for network congestion than others.
So no I can't really see how would your approach would be any better, the approach that MSFT took was quite good, netflix, wikipedia, youtube, facebook etc. with what seems to be realistic user behaviour. What you want to do is to put in test that would produce fair results for fairness sakes that's not how you evaluate anything because it would not yield you any real world data.
Re: Why we challenge Microsoft’s battery test
#135Earlier quoted context omitted.
The way I read the blog is that the adblocking is enabled to show that even _with_ an expensive extension enabled it still beats Edge, but that was in my opinion not the gist of the post. The post was about that microsoft is not transparent about the methodology/setup/scripts/target websites used. A third party should be able to support Microsoft's claims. Is it possible for the setup to be published to your github.c…
Adblock isn't an "expensive extension". Ads are the expensive thing, so blocking them saves power.
I believe Firefox partly resolved the problem on their end later on, although I can't tell you the exact status of things. Nevertheless, there's definitely some precedence for claiming that ad blocking is an expensive operation. Intuitively, you'd think that network based blocking would be enough, but it won't work against same domain ad sources (a typical example being facebook ads), while css selectors are able to capture a bit more depth. Nevertheless, I don't know exactly what has happened since, although I do recall Firefox making some adjustments on their end that improved the situation.
[0] http://www.extremetech.com/computing/182428-ironic-iframes-a...
[1] https://blog.mozilla.org/nnethercote/2014/05/14/adblock-plus...
Re: Why we challenge Microsoft’s battery test
#136Edge platform team employee here. A couple thoughts on methodology, focusing on our scripted test (under "stay productive longer" at https://blogs.windows.com/windowsexperience/2016/06/20/more-... ), which is most comparable to Opera's test; nobody seems to challenge our Netflix results: * We did not enable ad blockers, because we were testing the efficiency of the network stack, browser rendering pipeline, etc.; if…
How did you control for the specific ads the browsers were served? Ads are highly dynamic. Unless you (can) include the differences in your methodology you better eliminate that influence.
Re: Why we challenge Microsoft’s battery test
#137Wow thanks for the moderation! I guess here in Hacker News we are all smart engineers and scientists that rely on the scientific method except for when it comes to microsoft when we all turn into drooling idol worshiping children. I wonder whether MS has a well orchestrated campaign to have multiple employees troll hn, or whether they just go to the source and buy moderation points in bulk from ycombinator.
It's not about the question, it's about the tone ;-)
Re: Why we challenge Microsoft’s battery test
#138Earlier quoted context omitted.
Ding, ding, ding and to be fair this is the problem in both methodologies. The only way to accurately perform this test is to spider a bunch of sites, save the contents to a locally hosted HTTPD and ensure all third-party JS calls are resolved locally and test both against the exact same sites as they were spidered at a point in time. You can simply not account for changes which may happen to the markup, ad network b…
You don't need to go to such complicated lengths. Just perform enough tests (as in, a statistically large enough amount) and a distribution will form. That also captures the variability of real world network effects.
Re: Why we challenge Microsoft’s battery test
#139Edge platform team employee here. A couple thoughts on methodology, focusing on our scripted test (under "stay productive longer" at https://blogs.windows.com/windowsexperience/2016/06/20/more-... ), which is most comparable to Opera's test; nobody seems to challenge our Netflix results: * We did not enable ad blockers, because we were testing the efficiency of the network stack, browser rendering pipeline, etc.; if…
Re: Why we challenge Microsoft’s battery test
#140Earlier quoted context omitted.
You don't need to go to such complicated lengths. Just perform enough tests (as in, a statistically large enough amount) and a distribution will form. That also captures the variability of real world network effects.
It's hardly complicated. I've put such tests together in an afternoon. In fact, whatever is added in complexity is gained by the fact fewer tests are necessary. Via this mechanism you can also remove any questions about compression, use of HTTP/2, etc., which could impact the tests based on server-side choices when it comes to serving data to either platform. Equal always equals better.