Invalid benchmark tests

[webbles]

I've been looking into why some browsers are faster than others on the PeaceKeeper tests and I've noticed a few issues in the benchmark code.

1 - All the fps tests use a settimeout of 1 ms (check the runFPSTestFrame function in benchmark.js).

All of the browsers limit how fast settimout runs. Chrome limits it to 4ms while Firefox and Safari limit it to 10ms. Opera used to limit it to 10ms but it looks like they've increased it. Check out http://code.google.com/p/chromium/issues/detail?id=792 and http://ejohn.org/blog/analyzing-timer-performance/ for information on the limit.

What that means is that Chrome will always be twice as fast as the other browsers just because of their lower limit. So, instead of testing how fast a browser can perform javascript, you are actually testing what the settimout limit is.

2 - Code to set the repeat limit to infinity on browsers other than IE is incorrectly written.

If you look in the init function in benchmark.js, you'll see a chunk of code that is supposed to make browsers other than IE run for a specific amount of time instead of a specific number of iterations. The code is below:

// Set test repeat limit to ~infinity on other browser than IE.
// IE counts operations, other browsers count time.
if (navigator.userAgent.toString().indexOf("MSIE") != -1) {
testOperationLimit = 99999999;
}
Two errors are apparent. First, the check is actually testing if the browser is IE (the != should be a ==). Second, the line inside the if statement should be benchmark.testOperationLimit = 99999999;. Otherwise, it is just setting a global variable that has no effect on the test (which is a good thing or else IE would be useless on the test).

This means that some of the benchmarks are performed so quickly (less than 50ms) that you start benchmarking how accurate the timer functions are in the browser.

[jaamo]

Hi webbles,

Thank you for your code review. You've found good points.

1. It's true that browser's have different limits for setTimeout function and we were aware of that when we designed tests. To eliminate the problem we created tests which rendered a single frame so slowly that time limits are not an issue. Of course computers and browsers get faster and faster. At some day all browsers are too fast for Peacekeeper and fps-tests became meaningless. Then we need to rebalance tests.

Maybe in the future we can relay on methods like requestAnimationFrame:
http://weblogs.mozillazine.org/roc/a...uestanima.html

There are also other methods to do animations:
http://ajaxian.com/archives/settimeout-delay


2. It seems that you've found a bug in code. We're looking into this.

[D.Jackson]

Hi webbles,

Thank you for your code review. You've found good points.

1. It's true that browser's have different limits for setTimeout function and we were aware of that when we designed tests. To eliminate the problem we created tests which rendered a single frame so slowly that time limits are not an issue. Of course computers and browsers get faster and faster. At some day all browsers are too fast for Peacekeeper and fps-tests became meaningless. Then we need to rebalance tests.

Maybe in the future we can relay on methods like requestAnimationFrame:
http://weblogs.mozillazine.org/roc/a...uestanima.html

There are also other methods to do animations:
http://ajaxian.com/archives/settimeout-delay


2. It seems that you've found a bug in code. We're looking into this.

So if I understand correctly, the current Peacekeeper test does not favor browsers like Chrome with shorter settimeout limits, because the settimeout time is an insignificant increment to the amount of time required to accomplish the entire Peacekeeper workload. Correct?

[webbles]

Hi webbles,

Thank you for your code review. You've found good points.

1. It's true that browser's have different limits for setTimeout function and we were aware of that when we designed tests. To eliminate the problem we created tests which rendered a single frame so slowly that time limits are not an issue. Of course computers and browsers get faster and faster. At some day all browsers are too fast for Peacekeeper and fps-tests became meaningless. Then we need to rebalance tests.

Maybe in the future we can relay on methods like requestAnimationFrame:
http://weblogs.mozillazine.org/roc/a...uestanima.html

There are also other methods to do animations:
http://ajaxian.com/archives/settimeout-delay

Actually, the time limits do matter. The community tests are heavily weighted to Chrome because of the settimeout. Chrome is twice as fast and only because it can run every 4 ms while FF4 waits every 10 ms. The first grid test is also affected as well as the ball test. I'll get you the full list of tests that are affected by it tonight.

Another factor that causes FF4 to be slower is that you request "new Date" every single iteration in both the FPS tests and the non FPS tests. On Windows XP, FF doesn't compute that value very fast compared to other browsers and systems. So, the test framework is causing Windows XP FF4 to look slower than it should.

[webbles]

Hi webbles,

Thank you for your code review. You've found good points.

1. It's true that browser's have different limits for setTimeout function and we were aware of that when we designed tests. To eliminate the problem we created tests which rendered a single frame so slowly that time limits are not an issue. Of course computers and browsers get faster and faster. At some day all browsers are too fast for Peacekeeper and fps-tests became meaningless. Then we need to rebalance tests.

Maybe in the future we can relay on methods like requestAnimationFrame:
http://weblogs.mozillazine.org/roc/a...uestanima.html

There are also other methods to do animations:
http://ajaxian.com/archives/settimeout-delay

It looks like my previous post didn't make it. So I'll try again.

The settimeout actually does affect the tests. Any test where Chrome is above 100 fps is going to make Chrome look a lot faster than it is. All the other browsers can't go over 100 fps just because of their built-in limit, not because they are slower. And there are a few tests that have Chrome over 100 fps.

And if you set the timeout to a more reasonable limit (like 10ms), Chrome stops being the fastest browser on the test. I'll figure out the lists of tests tonight, but if I remember correctly, the first grid test and all the community tests show this.

[webbles]

Actually, the time limits do matter. The community tests are heavily weighted to Chrome because of the settimeout. Chrome is twice as fast and only because it can run every 4 ms while FF4 waits every 10 ms. The first grid test is also affected as well as the ball test. I'll get you the full list of tests that are affected by it tonight.

I've ran the first 9 tests (render and community suite) with a 1ms and 10ms in Chrome7, FF4 and Safari 5 (on Windows XP).

Chrome7 is 1.47 times faster than itself with 1ms vs 10ms
FF4 is .97 times faster than itself with 1ms vs 10ms
Safari 5 is 1.01 times faster than itself with 1ms vs 10ms

Chrome7 is 2.08 times faster than FF4 at 1ms
Chrome7 is 1.46 times faster than FF4 at 10ms
Chrome7 is 1.84 times faster than Safari5 with 1ms
Chrome7 is 1.34 times faster than Safari5 with 10ms

Plus 4 of the tests, Chrome exceeded 100fps which the other two have no chance of duplicating.

This shows that the 1ms settimeout is favoring Chrome over the other browsers and skews the results. If you really want a fair comparison, you should limit it to 10ms.

The numbers are below:

                           1ms    1ms        1ms      10ms   10ms      10ms
                           FF4    Chrome    Safari    FF4    Chrome    Safari
render    renderChart    31.812    97.475    79.062    31.635    79.131    78.837
render    renderGrid01    82.811    197.12    90.927    81.906    92.132    90.824
render    renderGrid02    54.259    139.563    76.057    64.774    78.723    77.395
render    renderGrid03    5.016    13.317    3.542    4.961    13.34    3.527
render    renderPhysics    38.922    70.362    40.64    39.755    49.942    39.887
community    community01Encrypt    73.098    145.742    56.664    73.453    78.49    55.178
community    community02ParseXML    13.587    17.456    21.852    15.118    15.239    21.045
community    community03Filter    65.492    101.695    72.189    62.003    72.757    71.571
community    community04Sort    54.945    79.787    72.546    59.117    63.165    71.785

[webbles]

I've found another issue.

3 - arrayCombined tests array functions on an empty array

In the run function, you test concat, join, pop, shift, reverse, sort, slice, and splice. The functions pop, shift, and splice all modify the original array. So, after the first iteration, this.data500 is reduced to 200 items (the pop and shift reduce it to 400 and the splice reduces it to 200). The second iteration, it is reduced to 50 (pop and shift reduce it to 100 and the splice reduces it to 50). On the third iteration, half of the pop and shift loop reduces it to 0 and it stays at 0 from then on. So 9998 iterations are just testing how fast a browser can sort, revers, pop, shift, slice, and splice an empty array.

That is an interesting data point to time, but I doubt that is what you really wanted. You could fix it by also doing push, unshift, and splice (with the removed elements).

[D.Jackson]

Actually, the time limits do matter. The community tests are heavily weighted to Chrome because of the settimeout. Chrome is twice as fast and only because it can run every 4 ms while FF4 waits every 10 ms. The first grid test is also affected as well as the ball test. I'll get you the full list of tests that are affected by it tonight.

Another factor that causes FF4 to be slower is that you request "new Date" every single iteration in both the FPS tests and the non FPS tests. On Windows XP, FF doesn't compute that value very fast compared to other browsers and systems. So, the test framework is causing Windows XP FF4 to look slower than it should.

I have questions that look at the relevance of the timeout settings to browser performance from a different angle, and which deal with browser performance both in Peacekeeper and in browsing the web. Why does FF4 use a timeout that is more than twice as long as Chrome? If it gives FF4 a disadvantage in Peacekeeper, then this disadvantage may also show up as slower performance in some aspects of web browsing - these handicaps can't be a good thing for FF4.

It seems like there are two possibilities to explain this difference in timeout settings that immediately come to mind:

1. The timeout setting is somewhat arbitrary and FF4 code is capable of being changed to use a shorter timeout; a FF4 reconfigured in this way would then not have this timeout handicap and this would seem like a good and relatively easy improvement to make in FF4.
2. FF4 code actually requires this longer timeout because it does some aspect of web browsing less efficiently than Chrome, and as a consequence the FF4 timeout handicap is permanent. In this scenario, the longer timeout setting is an intrinsic requirement of the FF4 code structure, and Peacekeeper and other benchmarks would be quite right to test this limitation.

If it is possibility #1, then maybe the fix should be in FF4, not in Peacekeeper. If it is possibility #2, then FF4 has a bigger problem than slower Peacekeeper scores. However, I am definitely not a software expert and will look for better minds to set me straight.

[webbles]

I have questions that look at the relevance of the timeout settings to browser performance from a different angle, and which deal with browser performance both in Peacekeeper and in browsing the web. Why does FF4 use a timeout that is more than twice as long as Chrome? If it gives FF4 a disadvantage in Peacekeeper, then this disadvantage may also show up as slower performance in some aspects of web browsing - these handicaps can't be a good thing for FF4.

It seems like there are two possibilities to explain this difference in timeout settings that immediately come to mind:

1. The timeout setting is somewhat arbitrary and FF4 code is capable of being changed to use a shorter timeout; a FF4 reconfigured in this way would then not have this timeout handicap and this would seem like a good and relatively easy improvement to make in FF4.
2. FF4 code actually requires this longer timeout because it does some aspect of web browsing less efficiently than Chrome, and as a consequence the FF4 timeout handicap is permanent. In this scenario, the longer timeout setting is an intrinsic requirement of the FF4 code structure, and Peacekeeper and other benchmarks would be quite right to test this limitation.

If it is possibility #1, then maybe the fix should be in FF4, not in Peacekeeper. If it is possibility #2, then FF4 has a bigger problem than slower Peacekeeper scores. However, I am definitely not a software expert and will look for better minds to set me straight.

It's actually because of poorly designed websites. When Mozilla forked from Netscape, they didn't have a clamp. But that caused some websites to peg the CPU at 100% because of setTimeout(,0). So, they limited it to 10ms (https://bugzilla.mozilla.org/show_bug.cgi?id=123273) since IE and Netscape were doing that.

When Chrome came out, they also didn't have a clamp. But that again caused poorly designed websites to have issues. So they clamped it at 4ms (http://code.google.com/p/chromium/issues/detail?id=792).

Reading the HTML5 spec, though, does look like the clamp should be 4ms now (http://www.w3.org/TR/html5/timers.html#timers). I'll file a bug with Firefox to see if they'll increase it. I have no idea what Safari, IE, or Opera will do.

But, I still think that Peacekeeper should raise their limit. My reasons are:

1 - Peacekeeper is not testing compliance with HTML5 (which is the only one that specifies a specific clamp).
2 - It makes Chrome look extremely fast compared to other browsers when in fact it isn't that fast.
3 - It is in the framework code so it affects every single test. If they did a specific test that tested settimeout, then I'd be fine with it.

[webbles]

I've just found out that Opera 10.6 has a setTimeout limit of around 2ms. So, it looks like Opera is getting a bigger advantage than Chrome. So, Peacekeeper should raise the setTimeout to at least 4ms because that is what the HTML5 spec has it at.