Futuremark's Reply

Soon enough, I got a reply from Futuremark. Let's see what they had to say :-

The 52.16 drivers are approved to be used with 3DMark03 build 340, but with the exception of the theoretical PixelShader 2.0 test. As we pointed out, the PS2.0 score using 52.16 drivers is solely comparable between NVIDIA cards. The optimization is not approved by us, and that's why we pointed it out in the notes. All other tests & numbers (even the 3DMark score) are fully comparable with any other cards, assuming they also use approved drivers and build 340 of 3DMark03, of course.

We think it is better to have at least one driver available with which you can get reliable scores from most of the tests, than to have no driver available at all. Afterall, it is only the theroretical PS2.0 test which isn't comparable with other IHV's cards if using the 52.16 drivers, and that is pointed out in the notes.

Our testing procedure is something I can't really go into detail about.
We have different internal versions of 3DMark03 which we use whenever a new official WHQL driver is released. Of course we also take some screenshots to verify that the IQ is ok, and use some other tools if needed.. The benchmark tests are run using a group of different cards to ensure that there aren't any 3DMark03 specific optimizations for different cards (different DX class cards, different chip revisions etc).

So, according to Futuremark, only the Pixel Shader 2.0 test is invalid for comparison when you use their approved Rev. 52.16 driver. However, the last game test in 3DMark03, Mother Nature, uses Pixel Shader 2.0. So, wouldn't such an "optimization" affect the game test and the overall 3DMark score?

I asked Futuremark about this and here is their reply :-

In 52.16, the only optimization is present in the theoretical PS2.0 test, and it does not affect the game tests, nor the 3DMark03 overall score

Well, that is good to hear. But honestly, I would still prefer Futuremark to approve at least an alternative driver that does NOT come with such "optimizations". This will give people the choice of using the Rev. 52.16 driver or a completely cheat-free driver that will allow full comparison with other graphics cards.

In the interest of trustworthy and comparable results, I seriously think Futuremark should consider such a move. At the very least, it will allay suspicions that Futuremark and NVIDIA have something going on the sidelines.

As for the final word on the Rev. 53.03 driver, well, guess what. They already tested the Rev. 53.03 driver and found that :-

...with some display cards it does not fulfill the run rules. Please note that drivers may behave differently according to which display card they are used for.

In other words, Futuremark has determined that the NVIDIA Rev. 53.03 drivers did not follow their Optimization Guidelines for 3DMark (PDF) :-

1. It is prohibited to change the rendering quality level that is requested by 3DMark.
2. It is prohibited to detect 3DMark directly or indirectly. In its sole discretion, Futuremark may approve detection in order to fix a specified hardware error.
3. Optimizations that utilize the empirical data of 3DMark are prohibited.
4. Generic optimizations that do not violate the above rules and benefit applications in general are acceptable only if the rendering is mathematically consistent with that of Microsoft® DirectX® reference rasterizer.

In other words, Futuremark found that there was some hanky-panky going on in the Rev. 53.03 driver. In fact, they found that they saw "pretty big differences when comparing those two driver sets."

Just out of curiosity, I tested the Rev. 52.16 driver and compared it against the Rev. 53.03 driver, just to see how much the difference was. Let's go see the results!