Comparing benchmarks

Learn to compare benchmark results with benchstat.

Comparing benchmarks

A single run is noisy. For a correct comparison you run each version several times and compare them statistically with benchstat:

go test -bench=Ayristir -count=10 > eski.txt
# change the code
go test -bench=Ayristir -count=10 > yeni.txt
benchstat eski.txt yeni.txt

benchstat shows the average difference, the confidence interval and whether the difference is significant (p-value).

Reducing noise

Don't run other heavy work on the machine while measuring, keep a laptop plugged in (power saving lowers the frequency), and take at least 6–10 samples with -count. Storing benchmark results and comparing each change with benchstat in CI is a common way to catch performance regressions.

Exercise A simple comparison

Compare two joining methods: join 50 pieces with += and with strings.Builder. Run testing.Benchmark 5 times for each, take the smallest NsPerOp, and print whether the Builder is faster.

Show example solution

package main

import (
    "fmt"
    "strings"
    "testing"
)

var sink string

func enIyi(f func()) int64 {
    var min int64 = -1
    for k := 0; k < 5; k++ {
        r := testing.Benchmark(func(b *testing.B) {
            for i := 0; i < b.N; i++ {
                f()
            }
        })
        if min < 0 || r.NsPerOp() < min {
            min = r.NsPerOp()
        }
    }
    return min
}

func main() {
    arti := enIyi(func() {
        s := ""
        for i := 0; i < 50; i++ {
            s += "x"
        }
        sink = s
    })
    builder := enIyi(func() {
        var sb strings.Builder
        for i := 0; i < 50; i++ {
            sb.WriteString("x")
        }
        sink = sb.String()
    })
    fmt.Println(builder < arti)
}

Test yourself

What does it mean when benchstat shows "~"?
- The code got twice as fast
- The benchmark failed
- There are no allocations
- The difference isn't statistically significant
Show answer

The difference isn't statistically significant — "~" says the measured difference can't be told apart from noise; the change's performance effect isn't proven.

Repeat count

go test -bench=. ____=10 > yeni.txt

Show answer

go test -bench=. -count=10 > yeni.txt

Tool and role

-count=10

Many samples
benchstat

Statistical comparison
p-value

Significance
~

No significant difference

Next lesson: Summary - Benchmark

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies