Skip to content

fix(ci): compare new benchmarks with base#165

Open
Sahil-4555 wants to merge 3 commits intopk910:masterfrom
Sahil-4555:fix/benchmark-base-compare
Open

fix(ci): compare new benchmarks with base#165
Sahil-4555 wants to merge 3 commits intopk910:masterfrom
Sahil-4555:fix/benchmark-base-compare

Conversation

@Sahil-4555
Copy link
Copy Markdown
Contributor

@Sahil-4555 Sahil-4555 commented Apr 3, 2026

Issue:
The benchmark bot was not comparing the new NodeProveMulti/... and NodeProve/... benchmarks with base. The base run was using the older benchmark test harness, so base-lib.txt did not contain those benchmark names.

Fix:
This PR updates the benchmark workflow to sync benchmark test files from the PR checkout into the base checkout before running base benchmarks, only when those files differ. This lets the base run include the same benchmark names, so the bot can show proper comparisons without doing unnecessary sync work.

@codecov
Copy link
Copy Markdown

codecov bot commented Apr 3, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.75%. Comparing base (4f7b42e) to head (de8a699).

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #165   +/-   ##
=======================================
  Coverage   95.75%   95.75%           
=======================================
  Files          47       47           
  Lines       10921    10921           
=======================================
  Hits        10457    10457           
  Misses        290      290           
  Partials      174      174           
Components Coverage Δ
dynssz 99.61% <ø> (ø)
dynsszgen 92.21% <ø> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 3, 2026

Benchmark Results

Library Benchmarks

                                                        │ base-lib.txt │             pr-lib.txt             │
                                                        │    sec/op    │    sec/op     vs base              │
¹ need >= 6 samples for confidence interval at level 0.95
geomean                                                   179.3n         179.5n        +0.12%

                                                        │ base-lib.txt  │              pr-lib.txt               │
                                                        │     B/op      │     B/op       vs base                │
¹ need >= 6 samples for confidence interval at level 0.95
² all samples are equal
³ summaries must be >0 to compute geomean
geomean                                                               ³                  +0.04%               ³

                                                        │ base-lib.txt │             pr-lib.txt              │
                                                        │  allocs/op   │  allocs/op   vs base                │
¹ need >= 6 samples for confidence interval at level 0.95
² all samples are equal
³ summaries must be >0 to compute geomean
geomean                                                              ³                +0.00%               ³

Performance Benchmarks

                                          │ base-perf.txt │            pr-perf.txt             │
                                          │    sec/op     │    sec/op     vs base              │
Codegen_BlockMinimal/Unmarshal-4             56.37µ ± ∞ ¹   55.58µ ± ∞ ¹  -1.40% (p=0.008 n=5)
Codegen_BlockMinimal/UnmarshalReader-4       76.49µ ± ∞ ¹   75.48µ ± ∞ ¹  -1.32% (p=0.008 n=5)
Codegen_StateMinimal/UnmarshalReader-4       9.058m ± ∞ ¹   8.871m ± ∞ ¹  -2.07% (p=0.032 n=5)
Reflection_BlockMainnet/Unmarshal-4          56.58µ ± ∞ ¹   55.75µ ± ∞ ¹  -1.47% (p=0.032 n=5)
Reflection_StateMainnet/MarshalWriter-4      5.288m ± ∞ ¹   5.043m ± ∞ ¹  -4.63% (p=0.008 n=5)
Reflection_BlockMinimal/Unmarshal-4          56.80µ ± ∞ ¹   55.87µ ± ∞ ¹  -1.64% (p=0.008 n=5)
Reflection_BlockMinimal/UnmarshalReader-4    77.38µ ± ∞ ¹   75.88µ ± ∞ ¹  -1.94% (p=0.032 n=5)
Reflection_BlockMinimal/MarshalWriter-4      23.09µ ± ∞ ¹   23.31µ ± ∞ ¹  +0.93% (p=0.016 n=5)
Reflection_StateMinimal/MarshalWriter-4      4.565m ± ∞ ¹   4.470m ± ∞ ¹  -2.10% (p=0.008 n=5)
¹ need >= 6 samples for confidence interval at level 0.95
geomean                                      728.6µ         724.5µ        -0.56%

                                          │ base-perf.txt │           pr-perf.txt            │
                                          │     B/op      │     B/op       vs base           │
¹ need >= 6 samples for confidence interval at level 0.95
² all samples are equal
³ summaries must be >0 to compute geomean
geomean                                                 ³                  ?               ³

                                          │ base-perf.txt │             pr-perf.txt              │
                                          │   allocs/op   │  allocs/op    vs base                │
¹ need >= 6 samples for confidence interval at level 0.95
² all samples are equal
³ summaries must be >0 to compute geomean
geomean                                                 ³                 +0.00%               ³

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant