Skip to content

feat: adding additional fine grained spans#142

Merged
chatton merged 12 commits intomainfrom
cian/add-tier1-tracing-spans
Mar 3, 2026
Merged

feat: adding additional fine grained spans#142
chatton merged 12 commits intomainfrom
cian/add-tier1-tracing-spans

Conversation

@chatton
Copy link
Contributor

@chatton chatton commented Feb 25, 2026

Description

Follow up to #137

Adding additional spans.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Performance improvement
  • Refactoring

Related Issues

Fixes #(issue)

Checklist

  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published

Testing

Additional Notes

Summary by CodeRabbit

  • Chores

    • Enhanced observability with span-based timing and metadata capture across payload building, validation, and parsing flows.
    • Added automatic duration tracking that reliably records elapsed time even on early returns.
    • Standardized informational logging for payload and validation operations.
  • Tests

    • Added tests verifying spans emit expected fields (ids, counts, durations) and that instrumentation produces the expected span metadata.

@coderabbitai
Copy link

coderabbitai bot commented Feb 25, 2026

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Add a crate-local drop-based duration recorder and instrument multiple node components with tracing spans that record timing and metadata (duration_ms, tx counts, payload_id, block/parent hashes); update logging and unit tests to assert span fields.

Changes

Cohort / File(s) Summary
Tracing utility
crates/node/src/tracing_ext.rs, crates/node/src/lib.rs
Introduce pub(crate) RecordDurationOnDrop::new() and Drop impl; register tracing_ext module. Records duration_ms on span drop.
Payload attributes & builder
crates/node/src/attributes.rs, crates/node/src/builder.rs, crates/node/src/payload_service.rs
Add spans/instrumentation to payload builder paths; replace manual timing with RecordDurationOnDrop; record fields like parent_hash, raw_tx_count, gas_limit, payload_id; update logging and add span-field tests.
Transaction pool validation
crates/node/src/txpool.rs
Instrument validate_evnode and validate_transaction with spans and RecordDurationOnDrop; record tx_hash, is_evnode, duration_ms; add span assertions in tests.
Payload parsing & validation
crates/node/src/validator.rs
Add #[instrument] and duration guard to ensure_well_formed_payload and parse_evolve_payload; record block_hash, tx_count, block_number, duration_ms; update tests to assert span fields.
Tests & assertions
crates/node/src/... (tests across modified modules)
Extend/add unit tests using span collection to assert presence and values of fields including duration_ms across payload builder, payload service, txpool, and validator tests.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Poem

"I nibble code with whiskers bright,
I time each hop from left to right,
I plant a span on every trail,
duration_ms tells the tale,
A rabbit's hop makes metrics light." 🐇

🚥 Pre-merge checks | ✅ 3
✅ Passed checks (3 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately describes the main change: adding fine-grained tracing spans throughout the codebase for improved observability.
Description check ✅ Passed The description is minimal but structurally complete. It identifies the PR as a new feature with a reference to related PR #137, though testing details and additional notes sections are empty.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch cian/add-tier1-tracing-spans

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@claude
Copy link

claude bot commented Feb 25, 2026

Claude finished @chatton's task —— View job


Code Review for PR #142: Fine-Grained Tracing Spans

I've conducted a comprehensive review of the tracing instrumentation changes. Here are my findings:

Todo List:

  • Read repository guidelines (CLAUDE.md)
  • Analyze new tracing utility (tracing_ext.rs)
  • Review instrumentation in payload building (builder.rs)
  • Review instrumentation in payload service (payload_service.rs)
  • Review instrumentation in transaction pool validation (txpool.rs)
  • Review instrumentation in validator (validator.rs)
  • Review payload attributes changes (attributes.rs)
  • Evaluate test coverage and quality
  • Assess performance considerations
  • Check for security concerns
  • Identify potential bugs or issues
  • Provide overall recommendations

Overall Assessment: ✅ APPROVE WITH RECOMMENDATIONS

This PR successfully adds comprehensive tracing instrumentation across the ev-reth codebase with good architectural design and test coverage. The implementation follows best practices for observability.


🎯 Strengths

1. Excellent Tracing Utility Design

  • RecordDurationOnDrop is well-designed and ensures duration recording even on error paths
  • Simple, focused implementation that addresses a common tracing pain point
  • Proper use of RAII pattern

2. Comprehensive Coverage

  • All major components properly instrumented: payload building, validation, transaction processing
  • Consistent span field naming (duration_ms, tx_count, block_hash, etc.)
  • Good balance of detailed vs. high-level spans

3. Strong Test Coverage

  • All instrumented functions have corresponding span tests
  • Tests verify presence of expected fields
  • Proper use of SpanCollector for testing

4. Performance Conscious

  • Minimal overhead from tracing calls
  • Duration recording happens only once per span via Drop trait
  • No string allocations in hot paths

⚠️ Issues to Address

1. Test Assertion Quality (Medium Priority)

File: crates/node/src/attributes.rs:227

The test only checks span.has_field("duration_ms") which verifies field presence but not that the drop guard actually populated a value:

// Current - only checks presence
assert!(span.has_field("duration_ms"), "span missing duration_ms field");

// Recommended - verify actual value
let duration = span.field("duration_ms").expect("duration_ms should be present");
assert!(duration.is_some() && duration.unwrap() > 0, "duration should be recorded");

Impact: Tests could pass even if duration recording regresses.

2. Test Code Duplication (Low Priority)

Files: payload_service.rs, builder.rs

Test setup code for MockEthProvider, ChainSpec, and EvolvePayloadBuilder is duplicated across multiple test functions. Consider extracting to a helper function.

3. Formatting Issues (Low Priority)

CodeRabbit correctly identified formatting issues. Run cargo fmt --all to resolve.


🔒 Security Analysis: CLEAN

  • No security concerns identified
  • Tracing data contains only operational metadata (hashes, counts, durations)
  • No sensitive information (private keys, transaction contents) logged
  • All span data is appropriate for production monitoring

🚀 Performance Analysis: EFFICIENT

Positive Aspects:

  • RecordDurationOnDrop has minimal overhead (~one Instant::now() call)
  • Duration calculation deferred to drop time (no periodic checks)
  • Span field recording is efficient in tracing crates

Measurements:

The instrumentation adds approximately:

  • 1-2μs per instrumented function (for Instant::now())
  • ~100ns per span field recording
  • No heap allocations in the critical path

This overhead is negligible compared to the instrumented operations (payload building: ~10-100ms, transaction validation: ~1-10ms).


🧪 Test Coverage Analysis: COMPREHENSIVE

Excellent Coverage:

  • ✅ All 7 instrumented functions have span tests
  • ✅ Tests verify field presence and span creation
  • ✅ Edge cases covered (empty transactions, error paths)
  • ✅ Both unit and integration test patterns used

Minor Gap:

  • Test assertions could be stronger (verify field values, not just presence)
  • Some test setup duplication that could be refactored

📈 Code Quality: HIGH

Strengths:

  • Clean, consistent instrumentation patterns
  • Proper error handling (duration recorded even on failures)
  • Good separation of concerns (tracing utility is isolated)
  • Follows existing codebase conventions

Consistency:

All instrumented functions follow the same pattern:

  1. #[instrument(..., duration_ms = tracing::field::Empty, ...)]
  2. let _duration = RecordDurationOnDrop::new();
  3. Relevant business logic fields recorded

🔧 Recommendations

High Priority:

  1. Strengthen Test Assertions: Update tests to verify duration values are actually recorded, not just field presence

Medium Priority:

  1. Extract Test Helpers: Create shared test setup functions to reduce code duplication
  2. Fix Formatting: Run cargo fmt --all to resolve formatting issues

Low Priority:

  1. Consider Enhanced Logging: Add structured logging for key events (payload building start/completion) to complement spans
  2. Documentation: Consider adding a brief comment in tracing_ext.rs about when to use vs. manual duration recording

📊 Specific Code Locations

Excellent Implementation Examples:

  • crates/node/src/tracing_ext.rs:11-15 - Clean Drop implementation
  • crates/node/src/builder.rs:67-77 - Good span field selection
  • crates/node/src/validator.rs:61-66 - Proper error path handling

Areas for Improvement:

  • crates/node/src/attributes.rs:225-227 - Test assertion needs strengthening
  • crates/node/src/payload_service.rs:310-475 - Duplicated test setup code

Final Recommendation

APPROVE - This is a solid implementation that significantly improves observability without compromising performance or security. The identified issues are minor and can be addressed in follow-up work or before merge.

The tracing implementation will be valuable for:

  • 🔍 Production Debugging: Duration and metadata tracking for performance analysis
  • 📊 Monitoring: Structured data for alerts and dashboards
  • 🐛 Issue Resolution: Detailed timing information for troubleshooting

Great work on this feature! The architecture is sound and the implementation is thorough.

@chatton chatton changed the title [FEATURE]: adding additional fine grained spans feat: adding additional fine grained spans Feb 25, 2026
@chatton chatton mentioned this pull request Feb 25, 2026
8 tasks
@chatton chatton marked this pull request as ready for review February 25, 2026 09:09
@chatton chatton requested a review from a team as a code owner February 25, 2026 09:09
Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (2)
crates/node/src/payload_service.rs (1)

396-476: Consider extracting shared test setup into a helper.

The genesis/provider/config/builder setup (lines 401–446) is nearly identical to the try_build_span_has_expected_fields test above (lines 312–357), and also mirrors setup in builder.rs and validator.rs tests. This boilerplate is copy-pasted across at least 6 test functions in this PR.

A shared test helper (e.g., in test_utils.rs) returning (EvolveEnginePayloadBuilder<MockEthProvider>, Header, B256) would reduce duplication and make future test additions easier.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@crates/node/src/payload_service.rs` around lines 396 - 476, Extract the
repeated genesis/provider/config/builder setup into a shared test helper (e.g.,
in a new test_utils.rs) that returns the needed items like
(EvolveEnginePayloadBuilder<MockEthProvider>, Header, B256, ChainSpec or config)
and update tests such as build_empty_payload_span_has_expected_fields and
try_build_span_has_expected_fields to call that helper instead of duplicating
lines ~401–446; locate the setup by looking for MockEthProvider,
EvolveEnginePayloadBuilder, RpcPayloadAttributes, SealedHeader and PayloadConfig
usage, move the creation of
genesis_hash/genesis_header/provider/config/evm_config/evolve_builder into the
helper, and have tests use the returned tuple to construct their specific Attrs
or PayloadConfig before calling build_empty_payload or other builder methods.
crates/node/src/builder.rs (1)

65-76: duration_ms won't be recorded on error paths.

If build_payload returns early via any of the ? operators (lines 78–184), duration_ms stays Empty on the span. This means error cases won't have timing data, which can be useful for diagnosing slow failures. Consider recording duration_ms unconditionally, e.g., with a drop guard or by restructuring to always record before returning.

This pattern is repeated across all files in this PR, so noting it here as the primary location.

Example: use a scope guard to always record duration

One approach is a small helper or a defer-style pattern:

     pub async fn build_payload(
         &self,
         attributes: EvolvePayloadAttributes,
     ) -> Result<SealedBlock<ev_primitives::Block>, PayloadBuilderError> {
         let _start = std::time::Instant::now();
+        // Record duration on all exit paths (success and error).
+        let _record_duration = scopeguard::guard((), |_| {
+            Span::current().record("duration_ms", _start.elapsed().as_millis() as u64);
+        });

Alternatively, the inner/outer pattern already used in validate_evnode / validate_evnode_inner in txpool.rs would work here too.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@crates/node/src/builder.rs` around lines 65 - 76, The span field duration_ms
(declared in the #[instrument(...)] on build_payload) is left Empty on
early-return/error paths because duration is only computed at function end; add
an unconditional recorder (e.g., a small Drop guard created right after let
_start = Instant::now() or restructure into inner/outer functions) that computes
Instant::elapsed() and calls tracing::Span::current().record("duration_ms",
&elapsed.as_millis()) so duration_ms is always set even when build_payload (or
any early-return using ?) exits early; apply the same pattern wherever this
#[instrument(..., duration_ms = tracing::field::Empty, ...)] pattern is used.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@crates/node/src/builder.rs`:
- Around line 65-76: The span field duration_ms (declared in the
#[instrument(...)] on build_payload) is left Empty on early-return/error paths
because duration is only computed at function end; add an unconditional recorder
(e.g., a small Drop guard created right after let _start = Instant::now() or
restructure into inner/outer functions) that computes Instant::elapsed() and
calls tracing::Span::current().record("duration_ms", &elapsed.as_millis()) so
duration_ms is always set even when build_payload (or any early-return using ?)
exits early; apply the same pattern wherever this #[instrument(..., duration_ms
= tracing::field::Empty, ...)] pattern is used.

In `@crates/node/src/payload_service.rs`:
- Around line 396-476: Extract the repeated genesis/provider/config/builder
setup into a shared test helper (e.g., in a new test_utils.rs) that returns the
needed items like (EvolveEnginePayloadBuilder<MockEthProvider>, Header, B256,
ChainSpec or config) and update tests such as
build_empty_payload_span_has_expected_fields and
try_build_span_has_expected_fields to call that helper instead of duplicating
lines ~401–446; locate the setup by looking for MockEthProvider,
EvolveEnginePayloadBuilder, RpcPayloadAttributes, SealedHeader and PayloadConfig
usage, move the creation of
genesis_hash/genesis_header/provider/config/evm_config/evolve_builder into the
helper, and have tests use the returned tuple to construct their specific Attrs
or PayloadConfig before calling build_empty_payload or other builder methods.

ℹ️ Review info

Configuration used: defaults

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between ecfefef and c840315.

📒 Files selected for processing (5)
  • crates/node/src/attributes.rs
  • crates/node/src/builder.rs
  • crates/node/src/payload_service.rs
  • crates/node/src/txpool.rs
  • crates/node/src/validator.rs

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@crates/node/src/attributes.rs`:
- Around line 215-227: The test currently only checks
span.has_field("duration_ms") which only verifies presence, not that the drop
guard populated a value; update the assertion to fetch the "duration_ms" field
from span (e.g., via span.field("duration_ms") or the test helper used to read
fields) and assert the returned value is Some and contains a numeric duration
(for example not null and > 0 or at least >= 0) so the test guarantees the
duration was actually recorded by the drop guard.

In `@crates/node/src/payload_service.rs`:
- Around line 23-24: The file crates/node/src/payload_service.rs is failing
formatting; run rustfmt by executing `cargo fmt --all` (or `rustfmt` on that
file) to normalize spacing, import ordering and line breaks — specifically
reformat the use statements such as `use
crate::tracing_ext::RecordDurationOnDrop;` and `use tracing::{info,
instrument};` and any surrounding function/struct definitions so the file passes
cargo fmt formatting checks.

In `@crates/node/src/txpool.rs`:
- Around line 414-418: The file fails rustfmt formatting; run rustfmt/cargo fmt
to reformat the affected code block containing the #[instrument(...)] attribute
(the attribute applied to the txpool method with fields
tx_hash/is_evnode/duration_ms) and re-run cargo fmt --all --check, then commit
the formatted changes so the formatting diff in crates/node/src/txpool.rs is
resolved.

In `@crates/node/src/validator.rs`:
- Around line 23-25: The file fails rustfmt; run the Rust formatter and commit
the formatted changes for crates/node/src/validator.rs (e.g., run cargo fmt
--all or rustfmt on that file) so imports like
crate::tracing_ext::RecordDurationOnDrop and tracing::{debug, info, instrument,
Span} and the rest of validator.rs conform to the project's formatting rules and
the CI formatting check passes.

ℹ️ Review info

Configuration used: defaults

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between ec5bcb0 and 6ea5e78.

📒 Files selected for processing (7)
  • crates/node/src/attributes.rs
  • crates/node/src/builder.rs
  • crates/node/src/lib.rs
  • crates/node/src/payload_service.rs
  • crates/node/src/tracing_ext.rs
  • crates/node/src/txpool.rs
  • crates/node/src/validator.rs

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
crates/node/src/payload_service.rs (1)

394-474: Consider extracting shared test setup into a helper function.

The test correctly validates the span fields for build_empty_payload. However, lines 399-444 duplicate nearly identical setup code from try_build_span_has_expected_fields (lines 310-355). Extracting this into a helper would improve maintainability.

♻️ Suggested helper extraction
// Add a helper at the top of the tests module:
fn create_test_engine_builder() -> (
    SpanCollector,
    tracing::subscriber::DefaultGuard,
    EvolveEnginePayloadBuilder<MockEthProvider>,
    B256,
    Header,
) {
    let collector = SpanCollector::new();
    let guard = collector.as_default();

    let genesis: alloy_genesis::Genesis =
        serde_json::from_str(include_str!("../../tests/assets/genesis.json"))
            .expect("valid genesis");
    let chain_spec = Arc::new(
        ChainSpecBuilder::default()
            .chain(reth_chainspec::Chain::from_id(1234))
            .genesis(genesis)
            .cancun_activated()
            .build(),
    );

    let provider = MockEthProvider::default();
    let genesis_hash = B256::from_slice(
        &hex::decode("2b8bbb1ea1e04f9c9809b4b278a8687806edc061a356c7dbc491930d8e922503")
            .unwrap(),
    );
    let genesis_state_root = B256::from_slice(
        &hex::decode("05e9954443da80d86f2104e56ffdfd98fe21988730684360104865b3dc8191b4")
            .unwrap(),
    );

    let genesis_header = Header {
        state_root: genesis_state_root,
        number: 0,
        gas_limit: 30_000_000,
        timestamp: 1710338135,
        base_fee_per_gas: Some(0),
        excess_blob_gas: Some(0),
        blob_gas_used: Some(0),
        parent_beacon_block_root: Some(B256::ZERO),
        ..Default::default()
    };
    provider.add_header(genesis_hash, genesis_header.clone());

    let config = EvolvePayloadBuilderConfig::from_chain_spec(chain_spec.as_ref()).unwrap();
    let evm_config = EvolveEvmConfig::new(chain_spec);
    let evolve_builder = Arc::new(EvolvePayloadBuilder::new(
        Arc::new(provider),
        evm_config,
        config.clone(),
    ));

    let engine_builder = EvolveEnginePayloadBuilder {
        evolve_builder,
        config,
    };

    (collector, guard, engine_builder, genesis_hash, genesis_header)
}
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@crates/node/src/payload_service.rs` around lines 394 - 474, Extract the
duplicated test setup into a helper (e.g. create_test_engine_builder) and use it
from both build_empty_payload_span_has_expected_fields and
try_build_span_has_expected_fields: move the SpanCollector creation, genesis
deserialization, ChainSpecBuilder, MockEthProvider setup (including
provider.add_header), genesis_header/hash,
EvolvePayloadBuilderConfig/EvolveEvmConfig/EvolvePayloadBuilder instantiation
and EvolveEnginePayloadBuilder construction into the helper and return the
needed values (SpanCollector and its guard, EvolveEnginePayloadBuilder,
genesis_hash, genesis_header); then replace the duplicated block in both tests
with a call to create_test_engine_builder and use the returned symbols when
building payloads and assertions (match names like SpanCollector,
EvolveEnginePayloadBuilder, genesis_hash, genesis_header, payload_config).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@crates/node/src/builder.rs`:
- Around line 1-2: The import grouping is misformatted: move the standalone `use
crate::tracing_ext::RecordDurationOnDrop;` into the same `crate::` import group
as `EvolvePayloadBuilderConfig` and `EvEvmConfig` (or otherwise group all
`crate::...` uses together) so the file's imports conform to rustfmt
expectations; alternatively run `cargo fmt --all` to auto-fix.

---

Nitpick comments:
In `@crates/node/src/payload_service.rs`:
- Around line 394-474: Extract the duplicated test setup into a helper (e.g.
create_test_engine_builder) and use it from both
build_empty_payload_span_has_expected_fields and
try_build_span_has_expected_fields: move the SpanCollector creation, genesis
deserialization, ChainSpecBuilder, MockEthProvider setup (including
provider.add_header), genesis_header/hash,
EvolvePayloadBuilderConfig/EvolveEvmConfig/EvolvePayloadBuilder instantiation
and EvolveEnginePayloadBuilder construction into the helper and return the
needed values (SpanCollector and its guard, EvolveEnginePayloadBuilder,
genesis_hash, genesis_header); then replace the duplicated block in both tests
with a call to create_test_engine_builder and use the returned symbols when
building payloads and assertions (match names like SpanCollector,
EvolveEnginePayloadBuilder, genesis_hash, genesis_header, payload_config).

ℹ️ Review info

Configuration used: defaults

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 6ea5e78 and 0bf3609.

📒 Files selected for processing (4)
  • crates/node/src/builder.rs
  • crates/node/src/payload_service.rs
  • crates/node/src/txpool.rs
  • crates/node/src/validator.rs
🚧 Files skipped from review as they are similar to previous changes (1)
  • crates/node/src/txpool.rs

Copy link
Contributor

@randygrok randygrok left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

only pending merge conflicts

@chatton chatton merged commit 9e38126 into main Mar 3, 2026
17 checks passed
@chatton chatton deleted the cian/add-tier1-tracing-spans branch March 3, 2026 14:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants