Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Huawei caught cheating performance test for new phones

techcrunch.com

Post reply on HN
Print view View original

1–10 of 126 posts

Re: Huawei caught cheating performance test for new phones

#3

Post by lysp »

> For the Huawei case, the rules are actually a little fuzzy. Phones are permitted to adjust performance based on workload, which results in peaks or dips in performance for different apps, but they are not permitted to hard-code peaks in performance specifically for the benchmark app. Huawei reportedly claimed that the peak in performance seen during the run of the benchmark app was an intuitive jump determined by AI; however, when an unlabeled version of the benchmark test was run, the phones were unable to recognize it and, as a result, displayed lower performances.

Source code for their AI engine:

if (app == benchmark) { increasePerformance(); }

Re: Huawei caught cheating performance test for new phones

#6

Post by halflings »

komaromy wrote: post #4

I don't see why an earnest benchmarking firm would let manufacturers know details like the name of the testing app.

Transparency. Everybody can run the benchmark on their own device [1] and check the results, that way they can't deny the results saying that it's malicious, a conflict of interest or a configuration error.

[1] https://play.google.com/store/apps/details?id=com.glbenchmar...

Re: Huawei caught cheating performance test for new phones

#7

Post by vvanders »

lysp wrote: post #3

> For the Huawei case, the rules are actually a little fuzzy. Phones are permitted to adjust performance based on workload, which results in peaks or dips in performance for different apps, but they are not permitted to hard-code peaks in performance specifically for the benchmark app. Huawei reportedly claimed that the peak in performance seen during the run of the benchmark app was an intuitive jump determined by A…

As someone who spent a ton of time in the smartphone GPU/CPU space from '09-'13 we saw every single vendor do this.

They'll happily ignore thermals, run custom shader microcode and any other tricks that'll boost their benchmark scores on common apps.

We always kept a set of internal benchmarks that we didn't share with vendors and used that as a part of SoC evaluation. It had the benefit of not being gamed and also lined up with our normal workloads so we had a decent idea of where we expected the whole system to land.

Re: Huawei caught cheating performance test for new phones

#8

Post by userbinator »

lysp wrote: post #3

> For the Huawei case, the rules are actually a little fuzzy. Phones are permitted to adjust performance based on workload, which results in peaks or dips in performance for different apps, but they are not permitted to hard-code peaks in performance specifically for the benchmark app. Huawei reportedly claimed that the peak in performance seen during the run of the benchmark app was an intuitive jump determined by A…

What exactly does "adjust performance" even mean? If it's something like frequency throttling, there should be a user setting to force maximums, even at the expense of battery life, and likewise another option to force minimums.
Post reply on HN