UL Delists Huawei Devices Due To Inaccurate Benchmarks

A few days ago we published our article addressing Huawei and Honor’s inaccurate benchmarking behaviour. In a nutshell, we had found out that this year’s devices had introduced a new thermal limiting behaviour that quickly throttled power consumption to ~4-4.5W in 3D workloads. While this in itself wasn’t an issue, the problem is that the firmware did not apply this new behaviour to a specific list of whitelisted 3D benchmarks.

Sidenote: Earlier this year, Futuremark (The 3DMark creators) had been acquired by UL, joining the global safety consulting and certification company as a benchmarking division.

We work closely with all benchmark vendors, and UL isn’t an exception. We had first given UL note of the behaviour two weeks ago and had been sharing our early results with the development team. Today UL published their independent confirmation of our results on their own devices, and have subsequently decided to remove the tested devices from their results database.

UL stresses that the kind of detection and optimization performed by Huawei infringes the company’s rules for manufacturers. Unfortunately this isn’t the first instance of a vendor being delisted, as most famously a slew of phone manufacturer had been delisted in 2013 after a more in-depth investigation of ours resulted in quite embarrassing results for a lot of vendors.

The UL team further explains that they’ve also been in touch with Huawei, and that the Chinese vendor is planning to address the behaviour by introducing a new “performance mode” that disables the new thermal throttling behaviour. In essence this mode would revert back to the behaviour we’ve seen in the past such as the Mate 9 – where the SoC is allowed higher peak performance figures at a cost of high power.

While this is still problematic for representative benchmarking for devices with Kirin 960’s and 970’s (due to the very large peak power at the highest performance states of those SoCs), I expect this to be a non-issue for the new Kirin 980, as it’s projected to have significantly better sustainable peak power figures. We're still looking forward to the new SoC in the upcoming Mate 20, and believe it to be an outstanding performer.