Skip to content

make execution index more compact#645

Merged
pk910 merged 6 commits intomasterfrom
pk910/remove-duplicate-txhash
Apr 17, 2026
Merged

make execution index more compact#645
pk910 merged 6 commits intomasterfrom
pk910/remove-duplicate-txhash

Conversation

@pk910
Copy link
Copy Markdown
Member

@pk910 pk910 commented Apr 17, 2026

Summary

Introduces tx_uid — a compact 8-byte transaction identifier that replaces the redundant (block_uid, tx_hash) composite key in execution indexing tables, significantly reducing storage and index overhead.

tx_uid encoding

tx_uid = block_uid << 16 | tx_index
       = (slot << 16 | block_unique_index) << 16 | tx_position
  • block_uid recoverable as tx_uid >> 16
  • tx_index recoverable as tx_uid & 0xFFFF
  • slot recoverable as tx_uid >> 32
  • Max slot ~2.1B (signed BIGINT) = 817+ years on mainnet at 12s slots

Schema changes

el_transactions — keeps block_uid, adds tx_uid, drops tx_index:

  • tx_uid added as a UNIQUE indexed column (block_uid << 16 | tx_index)
  • tx_index dropped (derivable from tx_uid & 0xFFFF)
  • Primary key remains (block_uid, tx_hash) for efficient block-based and hash-based lookups

el_event_index(block_uid, tx_hash, event_index)(tx_uid, event_index):

  • Drops block_uid (8 bytes) and tx_hash (32 bytes)
  • Replaces with tx_uid (8 bytes)
  • Saves 32 bytes per row

el_token_transfers(block_uid, tx_hash, tx_idx)(tx_uid, tx_idx):

  • Drops block_uid (8 bytes), tx_hash (32 bytes), and tx_pos (4 bytes)
  • Replaces with tx_uid (8 bytes)
  • Saves 36 bytes per row

el_transactions_internal(block_uid, tx_hash, tx_callidx)(tx_uid, tx_callidx):

  • Drops block_uid (8 bytes) and tx_hash (32 bytes)
  • Replaces with tx_uid (8 bytes)
  • Saves 32 bytes per row

Storage reduction estimate

Assumptions: mainnet-scale, 200 txs/block, ~2.5 events/tx, ~4 token transfers/tx, ~6.5 internal calls/tx.

Estimates include heap data + all associated indexes (PK + secondary). Dependent tables see large reductions because the 32-byte tx_hash and 8-byte block_uid are replaced by a single 8-byte tx_uid in both row data and every index entry.

Table Row data PK key size Removed indexes Reduction
el_event_index -32 bytes/row 44 → 12 bytes tx_hash_idx ~47%
el_token_transfers -36 bytes/row 44 → 12 bytes tx_hash_idx, block_uid_idx ~42%
el_transactions_internal -32 bytes/row 44 → 12 bytes tx_hash_idx ~47%
el_transactions +4 bytes/row unchanged swapped block_uid_idx → tx_uid_idx ~0%

The three dependent tables hold ~93% of all execution index rows (events, transfers, and internal calls vastly outnumber transactions). The weighted overall reduction across all four tables is approximately ~45%.

Query changes

  • Dependent table queries now use tx_uid instead of (block_uid, tx_hash) for all lookups
  • ORDER BY block_uid DESC, tx_index DESCORDER BY tx_uid DESC (equivalent ordering)
  • Address page handlers batch-fetch tx_hash from el_transactions via GetElTransactionsByTxUids() when needed for display
  • Cleanup uses WHERE tx_uid < threshold for dependent tables (threshold = blockUidThreshold << 16)

Migration

The migration (20260417000000_tx-uid) performs the following in a single transaction:

  1. Adds tx_uid column to el_transactions and populates it
  2. Recreates each dependent table with tx_uid via JOIN, migrating existing data
  3. Drops tx_index from el_transactions
  4. Rebuilds all affected indexes

Both PostgreSQL and SQLite migrations are included.

Test plan

  • Run migration on a database with existing execution index data
  • Verify transaction detail page shows correct data (events, transfers, internal txs tabs)
  • Verify address page shows correct token transfers and internal transactions
  • Verify slot page still enriches transactions with EL data
  • Verify new blocks are indexed correctly after migration
  • Verify retention cleanup still works (deletes old data by block threshold)
  • Verify reorg handling (orphaned blocks have distinct tx_uids)

@pk910 pk910 added the build-docker-image Automatically build docker image for PR branch label Apr 17, 2026
@pk910 pk910 merged commit 2bea73d into master Apr 17, 2026
14 checks passed
@pk910 pk910 deleted the pk910/remove-duplicate-txhash branch April 17, 2026 17:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build-docker-image Automatically build docker image for PR branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants