Samsung goes legit, stops cheating on benchmarks with latest Android update

They have decided to go legit. Finally.

Samsung Galaxy S4

Earlier this year, Samsung came under heavy fire for its benchmark shenanigans are dubious practices. The company deliberately built a whitelist of benchmark applications for products like the Note 3 and Galaxy S4. If any of these applications were detected, the CPU would immediately ramp up to 100% of its possible clock, regardless of the impact on power consumption or battery life.



This kicked Samsung’s benchmark scores into overdrive and predictably angered a number of reviewers and users. There was no chance that the behavior was incidental; investigations demonstrated that Samsung was deliberately detecting specific benchmarks rather than other applications, and handing those benchmarks different operating conditions than the rest of the programs that run on the device.

With Android 4.4 KitKat, Samsung has ceased this behavior. The whitelists have been removed, the browser no longer detects SunSpider or Rightware’s Browsermark, and the phone doesn’t display unusual activity or processor clocks when benchmark applications are run.

Why cheating matters

Ars Technica spoke with John Poole of Geekbench, who shared some pre- and post-patch data for the Galaxy S4 that neatly illustrates how Samsung twisted its performance results. Under Android 4.2.2, the Galaxy S4 scored an 1812 multicore score. That jumped to a 2114 using Android 4.3, but fell back to 1913 with Android 4.4.2 (which is confirmed as behaving normally).

Samsung Whitelist

In other words, Samsung inflated its multi-threaded benchmark scores by 11-16% depending on whether you consider the 1812 or 1913 as an appropriate baseline. The reason this kind of cheating matters is that it makes it impossible for reviewers or benchmark authors to fairly evaluate the product. Despite occasional claims to the contrary, the goal of a reputable reviewer is to build a suite of test software that collectively paints a fair picture of phone performance in real-world conditions.

Samsung’s whitelist watched for all of the most common smartphone benchmarks and deliberately altered the SoC’s performance in a way that didn’t reflect the experience of running games or other applications. Attempting to capture the performance of a device with a handful of applications and then translate those metrics to the variety of real-world conditions is already difficult; deliberate score inflation makes this nearly impossible.

We’re glad to see Samsung moving away from this behavior on older devices, but the real test will come with the launch of the Galaxy s5. If the company includes the same kind of whitelisted activity or attempts to implement a new type of sneakier obfuscation, the goodwill earned from stopping such practices today will go right out the window.

Source: VB Kid | WordPress

Your VB Kid
Psypher: