[FEAT] Add performance benchmarking and profiling infrastructure

[Stevengre]

Motivation

As the MIR semantics grows in complexity and test coverage expands (see #964), there is no systematic way to detect performance regressions or identify bottlenecks in symbolic execution. Issue #655 shows that performance problems have already surfaced (slow use functions), but we lack the tooling to investigate them systematically or to catch regressions automatically.

Proposed Feature

1. Benchmark Suite

Define a curated set of prove-rs test cases as a benchmark suite — a subset of the existing integration/data/prove-rs/ programs chosen to cover representative workloads (arithmetic, branching, enums, closures, iterators, etc.).

Each benchmark would record:

• Wall-clock time for kmir prove-rs
• Number of proof steps / rewrite rule applications (via K's --statistics output)
• Peak memory usage

Results are stored as a JSON artifact (e.g., bench-results.json) committed or uploaded as a workflow artifact for comparison.

2. Python-level Profiling via pyk.testing.Profiler

pyk already ships a cProfile-based Profiler class (pyk/testing/_profiler.py). We can integrate it into the pytest integration-test runner to generate .prof files for selected tests, enabling:

• Identification of hot Python functions in the SMIR→K transformation pipeline (kmir.py, smir.py)
• Easy investigation of issues like [INVESTIGATE] Why use functions slow down the symbolic execution #655 without manual instrumentation

A --profile pytest option (or a dedicated make profile-integration target) would enable this on demand.

3. GitHub Actions Workflow: benchmark.yml

A new workflow triggered on:

• push to master — baseline tracking
• workflow_dispatch — on-demand profiling for PRs under investigation
• Optionally: pull_request with a label like perf to gate on demand

Steps:

1. Build stable-mir-json + kmir (reuse Docker setup from test.yml)
2. Run the benchmark suite, capturing timing/step counts
3. Upload bench-results.json as a workflow artifact
4. On master push: compare against the previous baseline stored in a GitHub Actions cache and post a summary to the job summary ($GITHUB_STEP_SUMMARY)
5. Optionally: use benchmark-action/github-action-benchmark to track trends over time and comment on PRs when a regression threshold is exceeded

4. make benchmark Target

A local Makefile target for developers to run the benchmark suite locally and view a summary, mirroring CI behaviour.

Acceptance Criteria

• Benchmark suite defined (list of representative test cases with expected step counts)
• make benchmark target runs locally and outputs a timing/step-count table
• --profile mode generates .prof files for pytest integration tests using pyk.testing.Profiler
• benchmark.yml GitHub Actions workflow uploads benchmark artifacts on every master push
• CI posts a step-summary table comparing current vs. baseline results
• Documentation added to docs/dev/ on how to run benchmarks and interpret results

Related

• [INVESTIGATE] Why use functions slow down the symbolic execution #655 — slow use functions investigation
• Incrementally integrate external Rust test suites into mir-semantics CI #964 — external test suite integration (benchmark suite could overlap)
• pyk/testing/_profiler.py — existing profiler infrastructure available in the pyk dependency

[ehildenb]

@tothtamas28 can you point out any examples of profiling Python code that may be relevant here? Or profiling other semantics?

@jberthold maybe you can also post information about how we can profile the haskell backend directly too? It would be good to surface both levels of profiling to the test-suite directyl.

[tothtamas28]

@tothtamas28 can you point out any examples of profiling Python code that may be relevant here? Or profiling other semantics?

The pyk.testing module comes with a fixture that wraps cProfile.Profile. Here's an example of how it can be used in a test. Profiling results are written in the pytest temp directory under the specified filename.

pyk also has a dedicated test module and make target for profiling tests.

[jberthold]

@jberthold maybe you can also post information about how we can profile the haskell backend directly too? It would be good to surface both levels of profiling to the test-suite directyl.

To profile the HS Backend, I usually use cabal with a nix development shell (nix develop .#cabal) while I use stack for development (avoiding any accidental misconfigurations). In the nix develop shell , one can cabal configure --enable-profiling, which will write a cabal.project.local file that can be deleted afterwards and contains the profiling options. Then, the server is passed a runtime system option -p and possibly -po<file-stem> to control the output. This creates a *.prof file with a textual profiling summary.
Using Haskell profiling is probably much too low-level here. Also, the llvm interface conversions and the legacy backend typically dominate the profiling.

A slightly more high-level way to profile is to use logging option -l Timing for booster. It will provide timing information for each request, separating the implicit simplify requests for any fall-back during execute. It also writes (many!) timing records for LLVM backend evaluations. This can sometimes be enlightening, i.e. inspire thoughts about where time is spent during execute requests.

I like the ideas to have a benchmark suite and track performance trends from scheduled CI runs on master. The -l Timing information could be used (with the LLVM records removed) to inform about the performance of known requests. One could run a benchmark suite of bug reports. The timing log of the server is not strictly necessary for that, because kore-rpc-client has a command run-tarball to run requests from a provided tarball, and will also output timing information (including client-side time, but that one should be pretty invariable).

#655 — slow use functions investigation

Specifically about mir-semantics and this ticket, the interim simplifications are necessary because of a large amount of list updates to the <locals> which would accumulate during execution without simplifying in-between. The interim simplification keeps the unevaluated update thunks within a reasonable amount that can be handled when the server (for better or worse reasons) falls back to legacy kore.