Benchmarking Traps and Tricks

• Mismatched comparisons:

  • Competitor's machine slow due to software misconfiguration

• Extrapolation outside of data points:

  • Two runtime data points (8 and 32 processors) then linear extrapolation to 1024 processor machine

• Arithmetic sums to hide slower benchmarks

• Cheats for benchmarks:

  • Compilers looking for patterns in source
  • Inaccurate rendering for video benchmarks

Notes: