Benchmarking with Quake 4
July 12th 2006
1. Net Time Demos
A network demo is a recording of the network traffic during a multiplayer game. A network demo can be recorded at the a client connected to a multiplayer server. Such network demos are referred to as client netdemos. A network demo can also be recorded at the server. These network demos are referred to as server netdemos. ( see NetworkDemos )
When playing back a client netdemo Quake 4 will behave exactly as if playing a live game. Server netdemos record more information than client netdemos, and when playing back a server demo the view can be moved freely through the world or the view can be attached to any of the players that were playing on the server at the time of the recording.
Quake 4 allows playback of client netdemos as fast as possible where the engine continuously runs one game frame and one render frame. Playing back a network demo as fast as possible is referred to as a net time demo. Net time demos can be used to rate the overall performance of a system configuration. However, anyone using Quake 4 net time demos should be careful in properly using this tool for benchmarking.
First of all Quake 4 can be run in single or multi-threaded mode. When a system has multiple CPUs or a CPU with multiple cores the Quake 4 performance can be improved by enabling multi-threading (SMP) by setting the console variable r_useSMP to 1. However, when a system does neither have multiple CPUs nor a CPU with multiple cores the performance may not improve if multi-threading is enabled. Also make sure the lag-o-meter is disabled by setting the console variable net_clientLagOMeter to 0. The lag-o-meter aversely affects the performance when multi-threading is enabled. When enabling multi-threading in Quake 4 it is advised to disable multi-threading in the graphics driver. The multi-threading in Quake 4 generally outperforms the multi-threading in the graphics driver and once multi-threading in Quake 4 is enabled the multi-threading in the graphics driver may aversely affect the performance.
The net time demos reflect the overall system performance including CPU speed, memory speed and GPU speed. At any time one of the many components in a PC can become a bottleneck and limit the performance. As such correct interpretation of the results from net time demos requires identifying the bottlenecks in the system.
At higher resolutions the Quake 4 performance quickly becomes limited by the graphics card. This means that graphics card cannot keep up with the speed of the CPU and memory. On multi-processor or multi-core systems with the Quake 4 SMP code enabled the graphics card typically becomes more of a bottleneck because the load on the first cpu/core is reduced by moving work to the second cpu/core. To exclude the graphics card as a bottleneck the console variable r_skipRenderContext can be set to 1. With this console variable set to 1 all 3D rendering on the graphics card is skipped and the Quake 4 performance becomes limited only by the speed of the CPU, FSB, memory etc.
The Quake 4 performance is not only dependent on the CPU speed but also very much dependent on the memory speed. Increasing the memory speed can have close to the same effect on the performance as increasing the CPU speed. Quake 4 may not utilize all the available memory bandwidth but increasing the memory speed also lowers the memory access time which affects the performance.
The Quake 4 net time demos are meant to be used to rate the overall system performance. However, when using Quake 4 to benchmark the speed of graphics cards always make sure the system has the fastest CPU and the fastest memory available to make sure the CPU and memory subsystem have as little effect on the performance as possible.
When benchmarking the speed of CPUs always make sure the system has the fastest memory available to make sure the effect of the memory speed on the performance is as small as possible. Furthermore use a low resolution with the fastest graphics card solution available (SLI, Dual Cross Fire, etc.) and disable features like anti-aliasing. When benchmarking the speed of CPUs it is even better to set r_skipRenderContext to 1 to make sure the graphics card cannot be the bottleneck and does not affect the performance results.
When playing back a network demo, it is essential that the game code and assets are exactly the same as they were at the time of recording. If a lot of custom pak files were added to a Quake 4 installation it is very likely the filesystem needs to be reconfigured before it can play a demo with the right assets. First run a net time demo with the console variable demo_enforceFS set to 1 before doing any benchmarking. This will cause the filesystem to restart with the correct pak files and will also make sure all the right assets are cached and the performance is not affected by loading assets. To get the actual benchmark results set demo_enforceFS back to 0 and run the net time demo a couple of times and list the best result.
A net time demo can be started with the playNetTimeDemo console command. For benchmarking it is advised to use the demo001.netdemo that comes with Quake 4 ( version 1.2 and up ). This allows everyone to reproduce the same tests on the same hardware.
2. Do Not Use Render Demos
Quake 4 also supports so called "render demos". These render demos were used to test driver compatibility and are not meant for benchmarking. A render demo stores render commands and feeds them to the graphics driver. As such render demos do not reflect the performance of a system during actual gameplay because render demos push commands directly through the renderer and completely bypass the game code. In other words the game logic, physics etc. are not run when playing back a render demo.
The bottom line is that render demos do not reflect the performance during actual gameplay while net time demos do reflect the performance during actual gameplay. When playing a time demo with the playTimeDemo console command (as opposed to a net time demo with the playNetTimeDemo console command) Quake 4 plays a "render demo". In other words do not use time demos (playTimeDemo) but use net time demos (playNetTimeDemo) for benchmarking.