Futuremark Begins Delisting "Cheating" Mobile Devices from 3DMark Database
by Anand Lal Shimpi on November 26, 2013 9:37 PM EST- Posted in
- Smartphones
- Mobile
- Tablets
Over the summer, following a tip from @AndreiF7, we documented an interesting behavior on the Exynos 5 Octa versions of Samsung's Galaxy S 4. Upon detecting certain benchmarks the device would plug in all cores and increase/remove thermal limits, the latter enabling it to reach higher GPU frequencies than would otherwise be available in normal games. In our investigation we pointed out that other devices appeared to be doing something similar on the CPU front, while avoiding increasing thermal limits. Since then we've been updating a table in our reviews that keeps track of device behavior in various benchmarks.
It turns out there's a core group of benchmarks that seems to always trigger this special performance mode. Among them are AnTuTu and, interestingly enough, Vellamo. Other tests like 3DMark or GFXBench appear to be optimized for, but on a far less frequent basis. As Brian discovered in his review of HTC's One max, the list of optimization/cheating targets seems to grow with subsequent software updates.
In response to OEMs effectively gaming benchmarks, we're finally seeing benchmark vendors take a public stand on all of this. Futuremark is the first to do something about it. Futuremark now flags and delists devices caught cheating from its online benchmark comparison tool. The only devices that are delisted at this point are the HTC One mini, HTC One (One max remains in the score list), Samsung Galaxy Note 3 and Samsung Galaxy Tab 10.1 (2014 Edition). In the case of the Samsung devices, both Exynos 5 Octa and Qualcomm based versions are delisted.
Obviously this does nothing to stop users from running the benchmark, but it does publicly reprimand those guilty of gaming 3DMark scores. Delisted devices are sent to the bottom of the 3DMark Device Channel and the Best Mobile Devices list. I'm personally very pleased to see Futuremark's decision on this and I hope other benchmark vendors follow suit. Honestly I think the best approach would be for the benchmark vendors to toss up a warning splash screen on devices that auto-detect the app and adjust behavior accordingly. That's going to be one of the best routes to end-user education of what's going on.
Ultimately, I'd love to see the device OEMs simply drop the silliness and treat benchmarks like any other application (alternatively, exposing a global toggle for their benchmark/performance mode would be an interesting compromise). We're continuing to put pressure on device makers, but the benchmark vendors doing the same will surely help.
Source: Futuremark
Facebook
Twitter
RSS
46 Comments
View All Comments
KPOM - Wednesday, November 27, 2013 - link
I suppose Google/Motorola is "resorting to stunts" since they, also, don't cheat on their benchmarks?Cptn_Slo - Monday, December 2, 2013 - link
Samsung clock up their chips to appear higher in benchmarks without actually being faster in the real world. Intel actually clocks up in all applications.If understanding computing was a benchmark for your brain then it definitely did not clock up there.
Krysto - Wednesday, November 27, 2013 - link
The problem is Turbo-Boost is rarely activated in normal situations or for a very long time, so you won't benefit a lot from that Turbo-Boost anyway, but it does give Intel the opportunity to "win benchmarks", and also say that their chips are efficient, because they are usually only using the base clock speed.So Intel is having its cake and eating it, too, by using Turbo-Boost mainly in benchmark situations to show their chips are "high-performance", because even if it does use it say in games, it can't do it for more than a few minutes at a time, making it pretty useless anyway, while for a normal user it will mainly use the base clock speed, and have decent battery life BECAUSE it's actually using the low-performance version of the chip (i.e. not Turbo Boost).
Bottomline, Intel's chips are 99 percent of the time running in "power saving mode", and in benchmarks they run in "high-performance mode", which makes for very misleading benchmarks, because benchmarks are supposed to tell us how fast some chips are in GENERAL, not just for the 5 minutes the benchmark is running.
MrX8503 - Wednesday, November 27, 2013 - link
There is no problem with turbo boost. The reason turbo boost rarely ramps up is because users' workload is mostly light weight. Turbo boost will activate in real world heavy workloads, not just benchmarks. This is in no way the same as smartphone vendors boosting their clocks for benchmarks, but never for real world usage apps. Stop trying to give cheating vendors a pass.Tanclearas - Wednesday, November 27, 2013 - link
You have no idea what you're talking about. The base clock for my i7 is 2.4GHz, but when running things like Handbrake it will sustain boost clocks (typically close to 3.2GHz) for extended periods (over 20 minutes). I have seen Turbo boost work in other programs as well, and I don't have any "benchmark" programs on my computer at the moment.sundragon - Wednesday, November 27, 2013 - link
The issue is that Turbo Boost runs on any program that requires it, not just benchmarks... Samsung and HTC don't boost for other programs, they artificially boost for specific benchmark programs.Stop kissing their butts and admit this is a disservice to everyone and out right cheating.
Apple, fwiw, boots like Intel on all apps initially then drops to below the rated - If you Google Anadtech's benchmark you'd see it's uniform across all apps not just special special for benchmarks...
geniekid - Wednesday, November 27, 2013 - link
Turbo-Boost looks at process behavior and tries to optimize CPU operation for it. If the benchmark happens to meet the performance profile to activate TB, there's no problem with that. I disagree with your assertion that benchmarks are supposed to tell us how fast chips are "in general". I don't even know what you would classify as "general" behavior since I use my desktop 99% of the time for gaming - others don't. Even assuming benchmarks are supposed to measure "general" performance, it would be up to the benchmark program to generate "general" load.In the case of HTC/Samsung, they're not simulating some kind of load they know will cause the CPU to ramp up. They are literally looking to see if the application is Futuremark. That's something the benchmark program just can't get around, which is why it makes perfect sense to delist these devices.
Samsung/HTC, on the other hand, literally check to see if you are running Futuremark.
KPOM - Wednesday, November 27, 2013 - link
Some applications, such as video encoders, will cause Turbo Boost to keep running for as long as possible. Typing up an e-mail or editing a basic presentation isn't going to tax a CPU, so Turbo Boost won't stay on for very long unless it really needs to.What the OEMs are doing is detecting the benchmark, and then actively sending signals to the processors to remain on boosted or even overclocked speeds, rather than letting the CPU toggle normally as it would when a "real" application is running.
Cptn_Slo - Monday, December 2, 2013 - link
So you have no idea what turbo boost does.KoolAidMan1 - Friday, November 29, 2013 - link
Except that's not how normal CPU throttling works. The offending companies put flags to boost clock speed specifically for those benchmarks. If something else demands similar performance boosts, it doesn't do it. Intel and Apple don't cheat in this way, clock speed is regulated consistently.This is cheating, plain and simple. The baseless accusation at Apple in the end also makes your bias very very clear.