Comparing benchmarks
- Go course
- Intermediate
- Lesson 51/80
- 4 min
Learn to compare benchmark results with benchstat.
Comparing benchmarks
A single run is noisy. For a correct comparison you run each version several times and compare them statistically with benchstat:
go test -bench=Ayristir -count=10 > eski.txt
# change the code
go test -bench=Ayristir -count=10 > yeni.txt
benchstat eski.txt yeni.txtbenchstat shows the average difference, the confidence interval and whether the difference is significant (p-value).
Reducing noise
Don't run other heavy work on the machine while measuring, keep a laptop plugged in (power saving lowers the frequency), and take at least 6–10 samples with -count. Storing benchmark results and comparing each change with benchstat in CI is a common way to catch performance regressions.
Practice A simple comparison
Compare two joining methods: join 50 pieces with += and with strings.Builder. Run testing.Benchmark 5 times for each, take the smallest NsPerOp, and print whether the Builder is faster.
Starter code
package main
import (
"fmt"
"strings"
"testing"
)
func main() {
// buraya yaz
}Sample solution
package main
import (
"fmt"
"strings"
"testing"
)
var sink string
func enIyi(f func()) int64 {
var min int64 = -1
for k := 0; k < 5; k++ {
r := testing.Benchmark(func(b *testing.B) {
for i := 0; i < b.N; i++ {
f()
}
})
if min < 0 || r.NsPerOp() < min {
min = r.NsPerOp()
}
}
return min
}
func main() {
arti := enIyi(func() {
s := ""
for i := 0; i < 50; i++ {
s += "x"
}
sink = s
})
builder := enIyi(func() {
var sb strings.Builder
for i := 0; i < 50; i++ {
sb.WriteString("x")
}
sink = sb.String()
})
fmt.Println(builder < arti)
}Expected output
true
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 What does it mean when benchstat shows "~"?
- The code got twice as fast
- The benchmark failed
- There are no allocations
- The difference isn't statistically significant Correct answer
Explanation: "~" says the measured difference can't be told apart from noise; the change's performance effect isn't proven.
2 Repeat count
go test -bench=. ____=10 > yeni.txtgo test -bench=. -count=10 > yeni.txtExplanation: -count runs each benchmark the given number of times; benchstat computes statistics from these samples.
3 Tool and role
| Term | Meaning |
|---|---|
-count=10 |
Many samples |
benchstat |
Statistical comparison |
p-value |
Significance |
~ |
No significant difference |