Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Huawei caught cheating performance test for new phones

techcrunch.com

Post reply on HN
Print view View original

11–20 of 126 posts

Re: Huawei caught cheating performance test for new phones

#11

Post by remlov »

lysp wrote: post #3

> For the Huawei case, the rules are actually a little fuzzy. Phones are permitted to adjust performance based on workload, which results in peaks or dips in performance for different apps, but they are not permitted to hard-code peaks in performance specifically for the benchmark app. Huawei reportedly claimed that the peak in performance seen during the run of the benchmark app was an intuitive jump determined by A…

Reminds me of the time way back where Futuremark confirmed Nvidia was cheating on synthetic benchmarks by doing something similar except in reverse where they would degrade texture quality and use other tricks to artificially inflate scores in 3dmark. I'm certain ATI was caught doing so as well at some point.

Re: Huawei caught cheating performance test for new phones

#12

Post by MBCook »

vvanders wrote: post #7
lysp wrote: post #3

> For the Huawei case, the rules are actually a little fuzzy. Phones are permitted to adjust performance based on workload, which results in peaks or dips in performance for different apps, but they are not permitted to hard-code peaks in performance specifically for the benchmark app. Huawei reportedly claimed that the peak in performance seen during the run of the benchmark app was an intuitive jump determined by A…

As someone who spent a ton of time in the smartphone GPU/CPU space from '09-'13 we saw every single vendor do this. They'll happily ignore thermals, run custom shader microcode and any other tricks that'll boost their benchmark scores on common apps. We always kept a set of internal benchmarks that we didn't share with vendors and used that as a part of SoC evaluation. It had the benefit of not being gamed and also l…

Desktops have gone through it too: quack.exe and the HardOCP exposé. It’s been a long time since I thought of that.

Re: Huawei caught cheating performance test for new phones

#13

Post by jackconnor »

EZ-E wrote: post #9

I'm gonna get down voted but that's why I buy Apple phones. Comparing processors, ram, test scores means nothing compared to the actual experience

Yea I kind of agree. I think most people, myself included, just don't really recognize minor performance differences, only relatively large ones. And I've always thought that these things improve so fast year-to-year, that it's kind of pointless to try and keep up. So, i shoot for a phone I like that performs pretty well, and just leave it at that (hello iPhone SE).

Re: Huawei caught cheating performance test for new phones

#15

Post by Jerry2 »

vvanders wrote: post #7
lysp wrote: post #3

> For the Huawei case, the rules are actually a little fuzzy. Phones are permitted to adjust performance based on workload, which results in peaks or dips in performance for different apps, but they are not permitted to hard-code peaks in performance specifically for the benchmark app. Huawei reportedly claimed that the peak in performance seen during the run of the benchmark app was an intuitive jump determined by A…

As someone who spent a ton of time in the smartphone GPU/CPU space from '09-'13 we saw every single vendor do this. They'll happily ignore thermals, run custom shader microcode and any other tricks that'll boost their benchmark scores on common apps. We always kept a set of internal benchmarks that we didn't share with vendors and used that as a part of SoC evaluation. It had the benefit of not being gamed and also l…

>we saw every single vendor do this

Except Apple.

Re: Huawei caught cheating performance test for new phones

#17

Post by Shank »

userbinator wrote: post #8
lysp wrote: post #3

> For the Huawei case, the rules are actually a little fuzzy. Phones are permitted to adjust performance based on workload, which results in peaks or dips in performance for different apps, but they are not permitted to hard-code peaks in performance specifically for the benchmark app. Huawei reportedly claimed that the peak in performance seen during the run of the benchmark app was an intuitive jump determined by A…

What exactly does "adjust performance" even mean? If it's something like frequency throttling, there should be a user setting to force maximums, even at the expense of battery life, and likewise another option to force minimums.

No mobile device on the market runs at the same clock speed for any length of time. They’re always scaling down the clock speed for a variety of reasons (usually battery life and thermal management). If you ran at max speed all the time, you’d have trash battery life and probably cause overheating.

Re: Huawei caught cheating performance test for new phones

#18

Post by JoshTriplett »

userbinator wrote: post #8
lysp wrote: post #3

> For the Huawei case, the rules are actually a little fuzzy. Phones are permitted to adjust performance based on workload, which results in peaks or dips in performance for different apps, but they are not permitted to hard-code peaks in performance specifically for the benchmark app. Huawei reportedly claimed that the peak in performance seen during the run of the benchmark app was an intuitive jump determined by A…

What exactly does "adjust performance" even mean? If it's something like frequency throttling, there should be a user setting to force maximums, even at the expense of battery life, and likewise another option to force minimums.

> If it's something like frequency throttling, there should be a user setting to force maximums, even at the expense of battery life

Battery life is not the only concern; a device that sits in your hand or your pocket needs to carefully manage heat.

Re: Huawei caught cheating performance test for new phones

#19

Post by MBCook »

remlov wrote: post #11
lysp wrote: post #3

> For the Huawei case, the rules are actually a little fuzzy. Phones are permitted to adjust performance based on workload, which results in peaks or dips in performance for different apps, but they are not permitted to hard-code peaks in performance specifically for the benchmark app. Huawei reportedly claimed that the peak in performance seen during the run of the benchmark app was an intuitive jump determined by A…

Reminds me of the time way back where Futuremark confirmed Nvidia was cheating on synthetic benchmarks by doing something similar except in reverse where they would degrade texture quality and use other tricks to artificially inflate scores in 3dmark. I'm certain ATI was caught doing so as well at some point.

Yep. I believe ATI was the one where if you renamed quake.exe to quack.exe it suddenly performed significantly worse.

Re: Huawei caught cheating performance test for new phones

#20

Post by dmix »

MBCook wrote: post #12
vvanders wrote: post #7

Earlier quoted context omitted.

As someone who spent a ton of time in the smartphone GPU/CPU space from '09-'13 we saw every single vendor do this. They'll happily ignore thermals, run custom shader microcode and any other tricks that'll boost their benchmark scores on common apps. We always kept a set of internal benchmarks that we didn't share with vendors and used that as a part of SoC evaluation. It had the benefit of not being gamed and also l…

Desktops have gone through it too: quack.exe and the HardOCP exposé. It’s been a long time since I thought of that.

Backstory I found, I wasn't familiar with this story (2001).

https://www.hardocp.com/article/2001/10/23/optimizing_or_che...

TLDR: ATI did the same thing with Quake 3 Arena which was commonly used as a benchmark game for testing new video cards of that era and which many game news sites were using.

Post reply on HN