Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Huawei caught cheating performance test for new phones

techcrunch.com

Post reply on HN
Print view View original

31–40 of 126 posts

Re: Huawei caught cheating performance test for new phones

#31

Post by culturestate »

Someone1234 wrote: post #21
EZ-E wrote: post #9

I'm gonna get down voted but that's why I buy Apple phones. Comparing processors, ram, test scores means nothing compared to the actual experience

> Comparing processors, ram, test scores means nothing compared to the actual experience But Apple brags endlessly about their A11 "Bionic" chip and similar[0]. Arguing that Apple values experience over specs might have been a valid argument back with Steven Jobs, but Apple's presentations spend significant amounts of time talking about hardware specs these days. I don't specifically mind that Apple does, but people…

Apple has always tailored their marketing to whatever their advantage is at that moment. If they have the best tech, they'll talk about speeds and feeds - if they don't, they'll talk about the experience or some other aspect. This is almost as reliable a tick-tock cycle as Intel's.

Re: Huawei caught cheating performance test for new phones

#32

Post by vbezhenar »

StudentStuff wrote: post #24
Jerry2 wrote: post #15

Earlier quoted context omitted.

> we saw every single vendor do this Except Apple.

Apple plays this game, but in a different way. They know that apps like Geekbench test for only a short duration which won't necessarily hit their thermal throttling if the phone is cool enough. Detecting 100% load for a few dozen seconds and giving it full performance isn't hard, that is what Huawei and other vendors should have been doing. Issues like the Intel i9 performance going significantly downward with consi…

Well, short high load sounds like typical usage for some smartphone tasks. Like open a browser, load heavy website, check something and shutdown. If phone can boost CPU, I can load website faster. And if benchmarks want to test for a prolonged duration, surely they can just run longer?

Re: Huawei caught cheating performance test for new phones

#33

Post by kllrnohj »

dmix wrote: post #20
MBCook wrote: post #12

Earlier quoted context omitted.

Desktops have gone through it too: quack.exe and the HardOCP exposé. It’s been a long time since I thought of that.

Backstory I found, I wasn't familiar with this story (2001). https://www.hardocp.com/article/2001/10/23/optimizing_or_che... TLDR: ATI did the same thing with Quake 3 Arena which was commonly used as a benchmark game for testing new video cards of that era and which many game news sites were using.

That's not an entirely accurate TLDR here as what ATI did back then is now considered standard practice for GPU drivers and is welcome. It is a bit different than the mobile space as GPU drivers are doing game-specific optimizations rather than benchmark-specific ones. The end user experience is a net win for the major titles, and as long as there's no visual cost it's considered fair game.

It did go too far, even to the extreme of Nvidia hardcoding in their GPU driver the exact camera path that 3dmark took ( https://www.extremetech.com/computing/54154-driver-irregular... ), which is more like what the mobile space is grappling with now. That was solved through a combination of scolding and by just changing the benchmarks to be what people actually do. You don't see 3dmark show up in GPU reviews much, you instead see performance results in a handful of major games. If mobile follows suit and shifts to doing things like benchmarking the average/90th/99th percentile of a game of Fortnite or something then these cheating practices become less feasible as a whole.

Re: Huawei caught cheating performance test for new phones

#34

Post by dmoy »

StudentStuff wrote: post #29
jakevn wrote: post #26

Earlier quoted context omitted.

Are you implying that short burst performance is for gaming benchmarks?

Nope, not implying that at all. For some games that may make sense, but I think that'd fall under the sustained heavy CPU/GPU usage category for most games. Same for compiling large apps, interactive HTML5 PWAs and such.

I am now very confused about which usage of the term "gaming" y'all are referring to.

Re: Huawei caught cheating performance test for new phones

#35

Post by jakevn »

StudentStuff wrote: post #29
jakevn wrote: post #26

Earlier quoted context omitted.

Are you implying that short burst performance is for gaming benchmarks?

Nope, not implying that at all. For some games that may make sense, but I think that'd fall under the sustained heavy CPU/GPU usage category for most games. Same for compiling large apps, interactive HTML5 PWAs and such.

By gaming benchmarks, I mean falsifying results. Burst performance is the normal optimization for all CPUs and GPUs.

Re: Huawei caught cheating performance test for new phones

#36

Post by axaxs »

This comes as no surprise. This is a company who has also been caught claiming their phone took a photo - but left in DSLR exif data, claiming a phone had UFS but shipped EMMC instead, boasting to work with dev community - only to lock them out months later, among other shady practices. They make nice phones at a good price, but is not a company to be trusted.

Re: Huawei caught cheating performance test for new phones

#37

Post by vvanders »

Shank wrote: post #17
userbinator wrote: post #8

Earlier quoted context omitted.

What exactly does "adjust performance" even mean? If it's something like frequency throttling, there should be a user setting to force maximums, even at the expense of battery life, and likewise another option to force minimums.

No mobile device on the market runs at the same clock speed for any length of time. They’re always scaling down the clock speed for a variety of reasons (usually battery life and thermal management). If you ran at max speed all the time, you’d have trash battery life and probably cause overheating.

Each device has a pretty unique thermal profile(tablets dissipate heat better than small phones, more metal = better profile, etc) so these things tend to get tuned on a per-device basis.

We had one device I was helping bring up where you could run the GPU full-tilt for ~60s. After that the device would reboot and fail to come up until it had cooled enough. Was a fun process characterizing that device and our workloads in 60s bursts. Couple weeks later they had it tuned in where you could run it 24/7 as the thermal limits kicked in pretty aggressively.

Re: Huawei caught cheating performance test for new phones

#38

Post by bri3d »

MBCook wrote: post #12
vvanders wrote: post #7

Earlier quoted context omitted.

As someone who spent a ton of time in the smartphone GPU/CPU space from '09-'13 we saw every single vendor do this. They'll happily ignore thermals, run custom shader microcode and any other tricks that'll boost their benchmark scores on common apps. We always kept a set of internal benchmarks that we didn't share with vendors and used that as a part of SoC evaluation. It had the benefit of not being gamed and also l…

Desktops have gone through it too: quack.exe and the HardOCP exposé. It’s been a long time since I thought of that.

Shader replacement was extremely popular for years - see AMD's infamous Doom3 shader replacement: https://www.anandtech.com/show/1488/6 . NVidia drivers did the same thing although they were (generally) a bit smarter about it, relying on shader fingerprinting rather than application fingerprinting to prevent embarrassing "rename the EXE" expose articles. They were notoriously caught using a benchmark's camera path as a fingerprint: https://www.extremetech.com/computing/54154-driver-irregular... .

GPU drivers are still loaded with such replacements, either to work around bugs or to optimize games where a developer partnership was impossible or otherwise rejected. I think that with the shift to lower-level GPU APIs this practice is beginning to fade, but it's basically been par for the course since programmable shaders became a thing.

Re: Huawei caught cheating performance test for new phones

#39

Post by anyfoo »

userbinator wrote: post #8
lysp wrote: post #3

> For the Huawei case, the rules are actually a little fuzzy. Phones are permitted to adjust performance based on workload, which results in peaks or dips in performance for different apps, but they are not permitted to hard-code peaks in performance specifically for the benchmark app. Huawei reportedly claimed that the peak in performance seen during the run of the benchmark app was an intuitive jump determined by A…

What exactly does "adjust performance" even mean? If it's something like frequency throttling, there should be a user setting to force maximums, even at the expense of battery life, and likewise another option to force minimums.

If you enjoy a phone that runs hot, but only for a few minutes, depending on whether an empty battery or thermal protection kicks in first, sure.

Power management and thermal budgets are a very real thing. The times where all device’s components would run at full speed are (thankfully) over for quite some time.

20 years ago PCs had multiple incredibly loud constantly running fans, that I think hardly anyone would put up with anymore (and would obviously not work for phones anyway). Today’s phones are extremely more powerful than those PCs in many ways, and yet consume a fraction of energy per time unit. Does not work without complex power management.

Re: Huawei caught cheating performance test for new phones

#40

Post by lern_too_spel »

Jerry2 wrote: post #15
vvanders wrote: post #7

Earlier quoted context omitted.

As someone who spent a ton of time in the smartphone GPU/CPU space from '09-'13 we saw every single vendor do this. They'll happily ignore thermals, run custom shader microcode and any other tricks that'll boost their benchmark scores on common apps. We always kept a set of internal benchmarks that we didn't share with vendors and used that as a part of SoC evaluation. It had the benefit of not being gamed and also l…

> we saw every single vendor do this Except Apple.

https://www.theregister.co.uk/2003/06/24/apple_accused_of_ch...

http://benediktmeurer.de/2016/12/16/the-truth-about-traditio...

Post reply on HN