Improvement on Intel-PT-based fuzzing capabilities

[Evian-Zhang]

I noticed that LibAFL has been integrating the Intel-PT-based fuzzing capabilities using the ptcov crate. I have also implemented a full-featured Intel-PT decoder iptr using pure Rust with more idiomatic APIs in both low-level and bitmap-level APIs.

Evaluated on the famous libxdc_experiments benchmark, it turns out that iptr has comparable performance with libxdc, the fastest Intel PT decoder written in C, with each target completed in 5-10 seconds. However, ptcov cannot run most of the targets (possible due to the approach that I build up PtImage since libxdc dumps memory in page level instead of image level, leading to some instructions split into two pages), while some targets cannot complete in even 18 hours... I guess the reason for the low performance is that ptcov does not use TNT cache to accelerate the bitmap update.

I also managed to integrate iptr into LibAFL, creating a fuzzer with the same functionality with intel_pt_command_executor. Due to the fact that the target program is very small, the performance improvement is not as large as what we see in libxdc_experiments. Anyway, it can still discover the vulnerabilities as ptcov does, proving the full functionalities.

I wonder whether the LibAFL team is willing to accept iptr as an alternative Intel PT decoder. I can think of four feasible approaches:

1. iptr as an alternative with independent sub-crate like libafl_intelpt

2. iptr and ptcov being two configurable backend features in libafl_intelpt

   However, I found that the logics and APIs are highly coupled with ptcov in both libafl_intelpt and the Intel PT executor hook, with some ptcov-related structs exposed in public APIs.

   If we want to use features to select these two backends, some refactor works still need to be done.

3. Rewrite ptcov using iptr

4. Directly use iptr to replace ptcov.

I'm willing to assist in any of the above approaches. Which approach do you prefer? :)

cc @Marcondiro @domenukk

[Marcondiro]

Hi, very cool!

About the benchmark, did you publish the exepriment setup and the results anywhere? I'm very curious to check it out and also debug ptcov.

Cool that you managed to integrate your library with the command executor. Unfortunately it might not be the best example to showcase PT decoding performance changes since the bottleneck there is on process spawning. I plan to upstream an example targeting libpng (on the same line of the frida binary only fuzzer).

[Evian-Zhang]

Thank you for your quick reply! I'm outside and I'll put the benchmark code of ptcov in about 3-4 hours.

For the libpng thing, I would say that, unfortunately, my linux server will be shutdown about two weeks since tomorrow due to vacation, and I have no Intel machines available. After your finishing the use case, I'm willing to help testing iptr version as long as my linux server is brought back :)

[Evian-Zhang]

Benchmark codes

Benchmark code of ptcov:

`Cargo.toml`

[package]
name = "ptcov-bench"
version = "0.1.0"
edition = "2024"

[dependencies]
ptcov = "0.0.6"
clap = { version = "4", features = ["derive"] }
memmap2 = "0.9"

[profile.release]
lto = true
codegen-units = 1
strip = true

`src/main.rs`

use std::{fs::File, io::Read, path::PathBuf};

use clap::Parser;
use memmap2::Mmap;
use ptcov::PtImage;

#[derive(Parser)]
struct Args {
    #[arg(long)]
    input: PathBuf,
    #[arg(long)]
    page_addr: PathBuf,
    #[arg(long)]
    page_dump: PathBuf,
}

fn main() {
    let Args {
        input,
        page_addr,
        page_dump,
    } = Args::parse();
    let input = unsafe { memmap2::Mmap::map(&File::open(input).unwrap()).unwrap() };

    let page_dump_file = File::open(page_dump).unwrap();
    let mut page_addr_file = File::open(page_addr).unwrap();
    let page_addr_file_len = page_addr_file.metadata().unwrap().len();
    let num_pages = page_addr_file_len / 0x1000;

    let mut page_maps = Vec::with_capacity(num_pages as usize);
    let pages = unsafe { Mmap::map(&page_dump_file).unwrap() };
    let mut addr_buf = [0u8; 8];
    let mut offset = 0usize;
    while page_addr_file.read_exact(&mut addr_buf).is_ok() {
        let addr = u64::from_le_bytes(addr_buf);
        page_maps.push((addr & 0xFFFF_FFFF_FFFF_F000, offset));
        offset += 0x1000;
    }
    page_maps.sort_by_key(|(addr, _)| *addr);

    let mut pt_cov = ptcov::PtCoverageDecoderBuilder::new()
        .images(
            page_maps
                .into_iter()
                .map(|(addr, offset)| PtImage::new(pages[offset..(offset + 0x1000)].to_vec(), addr))
                .collect(),
        )
        .build()
        .unwrap();
    let mut bitmap = vec![0u8; 0x10000];
    pt_cov.coverage(&input, &mut bitmap).unwrap();
}

Compile the above code with

RUSTFLAGS="-C target-cpu=native" cargo build --release

And to compile benchmark code for iptr:

git clone --depth 1 https://github.com/Evian-Zhang/iptr && cd iptr
RUSTFLAGS="-C target-cpu=native" cargo build --release

Benchmark preparation

First, clone https://github.com/nyx-fuzz/libxdc_experiments. And in the experiments subdirectory, make sure in current PATH, python refers to python2 (you can use pyenv shell 2 to achieve this). Then run sh ./prepare_eval.sh to prepare benchmark set.

Then you should modify the run_eval.py like this:

`run_eval.py`

import subprocess
import json
import time

print("please run echo 1 | sudo tee /sys/devices/system/cpu/intel_pstate/no_turbo to reduce variance")
print("please run echo 0 | sudo tee /proc/sys/kernel/randomize_va_space to reduce variance")

EXPERIMENTS = [
    {
        "name": "mruby",
        "trace": "tmp_data//mruby_2GB",
        "image": "../test_data/mruby/data",
        "base":  0x400000,
        "end":   0x4b6000,
        "iters": 1000,
    },
    {
        "name": "unzip",
        "trace": "tmp_data//unzip_2GB",
        "image": "../test_data/Unzip/data",
        "base":  0x400000,
        "end":   0x427000,
        "iters": 1000,
    },
    {
        "name": "kafl",
        "trace": "tmp_data/kafl_eval_2GB",
        "image": "../test_data/kafl_eval/data.pagecache",
        "base":  0xffffffff83a00000,
        "end":    0xffffffff83a26160,
        "iters": 1000,
    },
    {
        "name": "foo",
        "trace": "tmp_data/foo_2GB",
        "image": "../test_data/foo/data",
        "base":  0x7ffff7819000,
        "end":   0x7ffff79ca000,
        "iters": 1000,
    },
    {
        "name": "kernel",
        "trace": "tmp_data/Kernel_2GB",
        "image": "../test_data/Kernel/data",
        "base":  0xffffffff81000000, 
        "end":   0xffffffff81e53ba0,
        "iters": 1000,
    },
    {
        "name": "avscript32",
        "trace": "tmp_data/avscript_2GB",
        "image": "../test_data/avscript_32/page_cache",
        "base":  0x1000,
        "end":   0xfffff000,
        "iters": 1000,
    },

    {
        "name": "infiniteloop1",
        "trace": "tmp_data/infinite_loop_linux1_2GB",
        "image": "../test_data/infinite_loop_linux/page_cache",
        "base":  0xffffffff81000000,
        "end":   0xffffffff81e54000,
        "iters": 1000,
    },
    {
        "name": "qemu",
        "trace": "tmp_data/qemu1_2GB",
        "image": "../test_data/qemu/page_cache",
        "base":  0x000555555554000,
        "end":   0x00555555d60000,
        "iters": 1000,
    },
]

TOOLS = [
    {
        "name": "libxdc",
        "path": "./libxdc/tester"
    },
    {
        "name": "libipt",
        "path": "./libipt_ptxed/tester"
    },
    {
        "name": "WinAFL",
        "path": "./winaflpt/tester"
    },
    {
        "name": "PTrix",
        "path": "./ptrix/tester"
    },
    {
        "name": "killerbeez",
        "path": "./killerbeez/tester"
    },
    {
        "name": "honggfuzz",
        "path": "./honggfuzz/tester"
    }
]

MY_TOOLS= [{"name": "iptr", "path": "/home/evian/iptr/target/release/libxdc-exp-one-round"}, {"name": "ptcov", "path": "/home/evian/ptcov-bench/target/release/ptcov-bench"}]

NUM_RUNS = 3


for exp in EXPERIMENTS:
    res = {}
    res[exp["name"]] = {}
    tt = res[exp["name"]]
    # for tool in TOOLS:
    #     tt[tool["name"]] = []
    #     tt_res = tt[tool["name"]]
    #     for run in range(NUM_RUNS):
    #         env = {
    #             "LD_LIBRARY_PATH":"./libipt/build/lib/:./xed/kits/xed-install-base/lib",
    #             "XDC_TRACE": exp["trace"],
    #             "XDC_IMAGE": exp["image"],
    #             "XDC_BASE" : str(exp["base"]),
    #             "XDC_END"  : str(exp["end"])
    #         }

    #         start = time.time()
    #         process = subprocess.run(["taskset", "-c", "0", tool["path"]], env=env, check=True)
    #         end = time.time()
    #         dur = end-start
    #         tt_res.append(dur)
    for tool in MY_TOOLS:
        tt[tool["name"]] = []
        tt_res = tt[tool["name"]]
        for run in range(NUM_RUNS):
            env = {
                "LD_LIBRARY_PATH":"./libipt/build/lib/:./xed/kits/xed-install-base/lib",
                "XDC_TRACE": exp["trace"],
                "XDC_IMAGE": exp["image"],
                "XDC_BASE" : str(exp["base"]),
                "XDC_END"  : str(exp["end"])
            }
            print(exp["trace"])

            start = time.time()
            try:
                process = subprocess.run(["taskset", "-c", "0", tool["path"], "--input", exp["trace"], "--page-dump", f'{exp["image"]}.dump', "--page-addr", f'{exp["image"]}.addr'], check=True)
            except:
                continue
            end = time.time()
            dur = end-start
            tt_res.append(dur)

    with open("results-mytools/results_%s.json"%(exp["name"]),"w") as f:
        json.dump(res, f)

Remember to change the real path in MY_TOOLS!

Then run the following command to run benchmark:

python3 ./run_eval.py

Notes for debugging ptcov

As I said before, the code for ptcov benchmark uses a not-so-good PtImage creation method. The libxdc's memory is dumped in page level instead of image level, which I think is more general in VM-level memory dump. The main difference for this is that you should consider the situation where one instruction is split into two pages. You can refer to libxdc's this code or iptr's this code.

I believe that ptcov is not that slow, it's just because of the PtImage construction. However, as the libxdc's experiments show, solutions without TNT cache (like the libipt target) are still much slower than fastest solutions. iptr-edge-analyzer follows similar TNT cache approach with libxdc, and has comparable performance. Adding TNT cache is not an easy work, and I would suggest our iptr-edge-analyzer to avoid re-inventing the wheels. Or if you want to add TNT cache in ptcov, I would suggest that you could consider build upon iptr-decoder, the low-level Intel PT packet handlers, which can be more de-coupled and has higher performance due to the optimized jump table.

Anyway, if you have any questions, I'm happy to help :)

[Marcondiro]

Thanks, I'll have a look.
Just one note, while it is true that the there is no "tnt cache"in ptcov, there is a caching system that avoids disassembly between two decision points (instructions requiring PT traces)

[Marcondiro]

Hi, I modified a bit your ptcov wrapper (published here https://github.com/Marcondiro/libxdc_experiments) and tuned ptcov a bit. So far the first 2 benchmarks work, I will investigate why the third fails (results are in the same libxdc_experiments fork, respectively ~1.15x, ~3.5x slowdown).
Are your iptr setup and results published somewhere? thanks :)

[Evian-Zhang]

Wow nice job! I'll publish the iptr's version of libxdc_experiments asap. Maybe tomorrow. The overall procedure has been described above.

[Evian-Zhang]

I have published the iptr's version of libxdc_experiments here Evian-Zhang/libxdc_experiments. However, I have no access to a Linux machine, and I cannot compile libxdc on my Mac, so the repo does not contain the detailed benchmark result. In fact, the overall slowdown is about 1.0x~1.3x in my previous test. (The slowest target for iptr is foo, which seems repeating some same TNT sequences a lot, which is very suitable for the libxdc's cache-optimization strategy).