Performance Tracking for GCC

This page contains daily performance results for SPEC95 using the GNU Compiler Collection (GCC). The main goals of this activity are:

Monitor the quality of code generated by GCC to avoid/fix performance regressions.
Get a realistic measure of the benefits we get from new improvements we make to GCC's optimization framework.

Ideally, we would like to use tools that give us an indication of how good/bad a job GCC is doing in specific areas of the optimization process. Unfortunately, SPEC is not suited for this. But it is an industry standard benchmark that can help us keep track of our progress. If you are interested in SPEC2000, check this page or Andreas Jaeger's page.

The system runs every day at 05:01 GMT. The source tree is updated, built from scratch and a validation run of SPECint and SPECfp is done. There is one chart for each individual SPEC benchmark that shows the daily results for base (-O2) and peak (-O3). Furthermore, for each day there is a directory with the following files:

Log file for the runspec command.
SPEC report in HTML, PostScript and ASCII formats.
Log file with the output of cvs up for that day.
Diff file with all the new ChangeLog entries for that day.

If you are interested in running SPEC on another platform, contact me to get a copy of the scripts. I will also appreciate any suggestions you may have about the data we collect, how we use it and what information we present on these charts.

Daily results for IA32

Performance charts for mainline GCC.

About the SPEC benchmark

SPEC marks are a speedup factor with respect to a base time for some reference machine. For instance, a typical result for 099.go

                     Base      Base      Base      Peak      Peak      Peak
   Benchmarks    Ref Time  Run Time     Ratio  Ref Time  Run Time     Ratio
   ------------  --------  --------  --------  --------  --------  --------
   099.go            4600     120        38.2      4600     116        39.6

The 'Ref Time' columns indicate how many seconds it takes for the reference machine to execute the benchmark. The 'Run Time' columns indicate the actual execution time on the test machine. The ratio between the two gives you the SPEC mark for that test. Higher numbers indicate better performance.

In our case, we are not interested in absolute marks but rather fluctuations in the results. Typically, a 2-5% fluctuation from day to day is to be expected. Sometimes we might even get wild variations on a single day (either positive or negative). What is interesting is when you see a peak or a depression that is maintained over time. That's when we take a look and see what's going on.

You can find more information about SPEC95 here. Note that SPEC95 has been officially retired. But the information we get from it is still useful.

Last modified: Nov 28, 2003