Invalid benchmark tests

[D.Jackson]

It's actually because of poorly designed websites. When Mozilla forked from Netscape, they didn't have a clamp. But that caused some websites to peg the CPU at 100% because of setTimeout(,0). So, they limited it to 10ms (https://bugzilla.mozilla.org/show_bug.cgi?id=123273) since IE and Netscape were doing that.

When Chrome came out, they also didn't have a clamp. But that again caused poorly designed websites to have issues. So they clamped it at 4ms (http://code.google.com/p/chromium/issues/detail?id=792).

Reading the HTML5 spec, though, does look like the clamp should be 4ms now (http://www.w3.org/TR/html5/timers.html#timers). I'll file a bug with Firefox to see if they'll increase it. I have no idea what Safari, IE, or Opera will do.

But, I still think that Peacekeeper should raise their limit. My reasons are:

1 - Peacekeeper is not testing compliance with HTML5 (which is the only one that specifies a specific clamp).
2 - It makes Chrome look extremely fast compared to other browsers when in fact it isn't that fast.
3 - It is in the framework code so it affects every single test. If they did a specific test that tested settimeout, then I'd be fine with it.

webbles, this is quite interesting and it sounds like it is an easily fixed design flaw in FF4. I hope your bug report gets their attention.

[Necroman]

Please update the tests somehow - they are terribly old. At least count now the "canvas test" results to the complete score - FF4, Chrome5(?)+, Opera 10, Safari 5 and now even IE9 supports canvas tag so let's include canvas performance to the overall results

This message is no longer valid:
"These tests use 'Canvas', a new web technology for drawing and manipulating graphics without external plug-ins. Canvas is not yet supported by all major browsers so the scores from these tests are not included in the overall Peacekeeper score. "

[D.Jackson]

Please update the tests somehow - they are terribly old. At least count now the "canvas test" results to the complete score - FF4, Chrome5(?)+, Opera 10, Safari 5 and now even IE9 supports canvas tag so let's include canvas performance to the overall results

This message is no longer valid:
"These tests use 'Canvas', a new web technology for drawing and manipulating graphics without external plug-ins. Canvas is not yet supported by all major browsers so the scores from these tests are not included in the overall Peacekeeper score. "

Well said, Necroman. Performance in the HTML5 multimedia and interactive functions is probably at least as important as javascript for determining how fast a browser feels on the modern web. The major browsers are all getting close to parity in javascript, but their HTML5 standards adherence and acceleration of web multimedia varies by factors of 10 (for example see Peacekeeper vs. graphics speed test results here).

Peacekeeper should definitely include the Canvas results in their scoring and it would be a great idea to also add in other HTML5 speed tests, including those testing GPU-acceleration. Perhaps highlighting browser abilities in a broader range of web functions will spur the various browser projects to fix their various shortcomings.

[stve]

The major browsers are all getting close to parity in javascript,

Don't be fooled by how close the Sunspider results are.
The Peacekeeper benchmark is designed to measure a browser’s JavaScript performance.
The overall result may be close between Opera & Chrome but if you examine the details there are large differences in the different sections.
Firefox & IE9 lag behind.

Probably some or maybe all browsers optimize their code with one eye on the benchmarks & most benchmarks are designed by Chrome , Safari, Firefox & IE.
If you write the test you select javascript that makes your browser look good compared to the competition.

[Jarnis]

Please update the tests somehow - they are terribly old. At least count now the "canvas test" results to the complete score - FF4, Chrome5(?)+, Opera 10, Safari 5 and now even IE9 supports canvas tag so let's include canvas performance to the overall results

This message is no longer valid:
"These tests use 'Canvas', a new web technology for drawing and manipulating graphics without external plug-ins. Canvas is not yet supported by all major browsers so the scores from these tests are not included in the overall Peacekeeper score. "

That statement is obviously showing the age of the test. We're actively considering options to update the benchmark to make it more useful with new browser generations - it is just that these things take some time and we hope to do more than just a quick tweak or two if scoring formula is going to be changed (ie. old scores would not be comparable any more with new ones).

[D.Jackson]

That statement is obviously showing the age of the test. We're actively considering options to update the benchmark to make it more useful with new browser generations - it is just that these things take some time and we hope to do more than just a quick tweak or two if scoring formula is going to be changed (ie. old scores would not be comparable any more with new ones).

I agree with your beliefs that benchmarks need to be thoughtfully constructed to be valuable, and that benchmarks need to be updated as the technology evolves.

I have suggestions of areas for improvement:

Speed:

• Javascript: Sunspider has become irrelevant with all browsers more or less fully optimized for this benchmark. On the other hand, Kraken, V8, JSBenchmark and Peacekeeper have continued to distinguish among the browsers according to javascript speed, and these tests sometimes give different answers as to which browser is fastest. Finding and combining the test elements that differentiate among browsers in these different benchmarks could lead to creation of a single javascript benchmark that is more universally valuable than any of the individual benchmarks.
• HTML5 speed: Canvas is tested (but not scored) in the current Peacekeeper. Canvas and many other HTML5 multimedia and interactive functions should be included to improve Peacekeeper.
• GPU-acceleration of HTML5 multimedia and interactive content: The IE9 Test Drive site has a wealth of HTML5 speed tests that show the truly vast advantage that GPU-acceleration gives to browsers that have this function. Currently only IE9beta and Firefox 4.0b10 and later have GPU-acceleration, and it makes them 100-fold faster on some of these speed tests (Speed Reading and Psychedelic Browsing, for examples, but many other tests show the same acceleration advantage and may test different subsets of HTML5 functionality). All of us would benefit if there were a GPU-acceleration component to a future Peacekeeper benchmark to highlight the speed advantages of browsers that have this functionality and to keep the heat on browser projects that have not yet implemented acceleration.
• Other?:

Standards support:

• Javascript: Sputnik was a comprehensive javascript test of standards support - all of the leading browsers do nearly perfectly on these tests, and so, like Sunspider, this may be an irrelevant measure to distinguish the best modern browser.
• HTML5: This is an emerging area with new standards still being added. However, the HTML5 Test does a good and fast job of measuring standards support for (currently) 400 HTML5 tests, and this HTML5 testing strongly differentiates among the various browsers (currently, Chrome leading and IE9 lagging far behind the pack).

Other important browser functions:

• Broad tests of browser capability: Browserscope implements broad testing of security, text display, javascript optimization support, multiple connections support, Acid3 and other tests of a variety of functions important for browser utility. Browserscope or something like it would be a good benchmark addition to to probe for broader capabilities important for day-to-day use.
• Memory use and management with single and multiple tabs: Tom's Hardware does such memory tests when they compare browsers, and such a measure of browser resource utilization would be good to incorporate into Peacekeeper.
• Multiple tab loading speed: Similarly, speed of opening large numbers of tabs containing a representative sample of modern internet content would also be a valuable benchmark.

I am sure I have left out many important functions that browsers will need to compete on, but this is a start.

[webbles]

The Peacekeeper benchmark is designed to measure a browser’s JavaScript performance.

That is not a very accurate description of peacekeeper. A better description is that it tests some HTML/Javascript interaction, canvas, a few regular expressions, a few array operations, and jquery's selector engine.

Some of the javascript that it tests is rarely used in tight loops and so the need to make it fast is not all that useful. One test just runs one regular expression over and over again (/.*star.*/i). I don't know why anyone would use that instead of the faster indexOf() function.

Also, if IE9's dead code detector worked better, the data and string functions would be completed eliminated since they are technically dead code.