Isolate feature doesn't work with `$ORIGIN` in `DT_RUNPATH`

[ComputerDruid]

While reading the blog post at https://harrystern.net/extrasafe-user-namespaces.html I thought to myself: that doesn't sound like it would work, because it would mess up how binaries find their shared libraries when they set a DT_RUNPATH with $ORIGIN in it.

I thought it would be quick to check this by whipping up a quick paired dylib + binary crate, but I was very wrong about how quick it would be.

But after hours of futzing with things, I managed to get a binary which the dynamic loader loads fine outside the isolated environment, but fails inside:

It goes like:

dylib/Cargo.toml

[package]
name = "dylib"
version = "0.1.0"
edition = "2021"
[lib]
crate-type = ["cdylib"]

dylib/src/lib.rs

#[repr(C)]
pub struct S {
    ptr: *const u8,
    len: usize,
}


#[no_mangle]
pub extern "C" fn hello() -> S {
    let s = "hello world!".as_bytes();
    let ptr = s.as_ptr();
    let len = s.len();
    S { ptr, len }
}

example/Cargo.toml

[package]
name = "example"
version = "0.1.0"
edition = "2021"

[dependencies]
dylib = { path = "../dylib"  }
extrasafe = { version = "0.5.0", features = ["isolate"] }

example/src/main.rs

use extrasafe::isolate::Isolate;
use std::collections::HashMap;

mod sys {

    #[repr(C)]
    pub(crate) struct S {
        pub ptr: *const u8,
        pub len: usize,
    }

    #[link(name = "dylib")]
    extern "C" {
        pub(crate) fn hello() -> S;
    }
}

fn hello() -> &'static str {
    // Safety: FFI with no special requirements
    let s = unsafe { sys::hello() };
    // SAFETY: hello returns a &'static str in its raw parts
    let s: &'static [u8] = unsafe { std::slice::from_raw_parts(s.ptr, s.len) };
    std::str::from_utf8(s).unwrap()
}

fn isolate_print(name: &'static str) -> Isolate {
    fn print() {
        println!("{}", hello());
    }
    Isolate::new(name, print)
}

fn main() {
    //works outside isolate:
    println!("{}", hello());
    Isolate::main_hook("isolate_print", isolate_print);
    let output = Isolate::run("isolate_print", &HashMap::new()).unwrap();
    dbg!(output);
}

Then we just need some final hackery to set up DT_RUNPATH:

cd example
cargo build
# make the loader look for libdylib.so in deps/ next to the executable (which happens to be where cargo puts it)
patchelf --set-rpath '$ORIGIN/deps' target/debug/example
target/debug/example

Which gives me:

hello world!
[src/main.rs:38:5] output = Output {
    status: ExitStatus(
        unix_wait_status(
            32512,
        ),
    ),
    stdout: "",
    stderr: "isolate_print: error while loading shared libraries: libdylib.so: cannot open shared object file: No such file or directory\n",
}

and looking inside strace -f I can see the working open at the top:

execve("target/debug/example", ["target/debug/example"], 0x7ffebd5fd9a8 /* 85 vars */) = 0
brk(NULL)                               = 0x55afdf5e8000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f763d642000
readlinkat(AT_FDCWD, "/proc/self/exe", "/home/cdruid/src/extrasafe_examp"..., 4096) = 63
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/home/cdruid/src/extrasafe_example/example/target/debug/deps/glibc-hwcaps/x86-64-v3/libdylib.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/home/cdruid/src/extrasafe_example/example/target/debug/deps/glibc-hwcaps/x86-64-v3/", 0x7ffda910c5c0, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/home/cdruid/src/extrasafe_example/example/target/debug/deps/glibc-hwcaps/x86-64-v2/libdylib.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/home/cdruid/src/extrasafe_example/example/target/debug/deps/glibc-hwcaps/x86-64-v2/", 0x7ffda910c5c0, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/home/cdruid/src/extrasafe_example/example/target/debug/deps/libdylib.so", O_RDONLY|O_CLOEXEC) = 3

followed by the broken one down below:

[pid 581730] execve("/proc/self/fd/3", ["isolate_print"], 0x55afdf5e8c00 /* 0 vars */ <unfinished ...>
...
[pid 581730] <... execve resumed>)      = 0
[pid 581730] brk(NULL)                  = 0x55c303abd000
[pid 581730] mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f908a71b000
[pid 581730] readlinkat(AT_FDCWD, "/proc/self/exe", "/memfd:isolate_memfd (deleted)", 4096) = 30
[pid 581730] access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
[pid 581730] openat(AT_FDCWD, "//deps/glibc-hwcaps/x86-64-v3/libdylib.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 581730] newfstatat(AT_FDCWD, "//deps/glibc-hwcaps/x86-64-v3/", 0x7ffd54011770, 0) = -1 ENOENT (No such file or directory)
[pid 581730] openat(AT_FDCWD, "//deps/glibc-hwcaps/x86-64-v2/libdylib.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 581730] newfstatat(AT_FDCWD, "//deps/glibc-hwcaps/x86-64-v2/", 0x7ffd54011770, 0) = -1 ENOENT (No such file or directory)
[pid 581730] openat(AT_FDCWD, "//deps/libdylib.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

Looking through man ld.so, it looks like the fix might be as simple as setting LD_ORIGIN_PATH based on dirname $(readlink /proc/self/exe)

[ComputerDruid]

Side note: I probably could have gotten this to work with a dylib instead of a cdylib, but I was really struggling to get it to work at all and figured cdylibs were probably more widely used. Even so I could never get -Crpath=yes to help here and had to resort to using patchelf to set DT_RUNPATH to what I wanted.

[boustrophedon]

Thanks for reporting this!
I will be the first to admit that while I know the basics of dynamic linkage, it is a black magic that I try not to dabble with.

Just to summarize:

• We're linking a library in at runtime (a reasonable thing to do and something you might want to use an Isolate for if it's proprietary)
• The binary is compiled/linked such that its rpath, which tells the runtime linker where to search for libraries, is using the special $ORIGIN variable which tells the dynamic linker to look relative to the current exe (that's the linker calling readlinkat on /self/proc/exe)
• After we exec, /self/proc/exe is a memfd so the linker doesn't know what to do and just outputs / when computing $ORIGIN
• The linker then tries to open /deps/<dylib> when looking for the dylib and fails because that doesn't exist.
• "Regular" dynamic libraries aren't affected because their path is essentially hardcoded

So I think that, as you mentioned, setting the LD_ORIGIN_PATH environment variable should probably work - did you try setting it in the Isolate::run call's env vars map? Incidentally I really should change the interface so that it only takes the isolate name and you can optionally add env vars like you can bind mounts with Isolate::new

Another option would be to bindmount the dylib to /deps/<dylib> if you know ahead of time you or some lib you're using is doing this, but at that point you might as well just fix it ahead of time with patchelf (i.e. un-dynamicize the rpath).

If LD_ORIGIN_PATH works, I think it would probably be best if I made Isolate::run return a builder and then you or I could add a method like enable_dylib_ld_origin() or something that's just a wrapper for setting LD_ORIGIN_PATH. Writing a test for it is going to be a pain and might be the impetus for splitting out isolate_test.rs into a separate crate.

[ComputerDruid]

"Regular" dynamic libraries aren't affected because their path is essentially hardcoded

Basically yes, it uses the system search path which is not relative to $ORIGIN. Only "bundled" libraries would be affected by the copy to the memfd.

But yes, your summary seems accurate.

Another option would be to bindmount the dylib to /deps/<dylib> if you know ahead of time you or some lib you're using is doing this, but at that point you might as well just fix it ahead of time with patchelf (i.e. un-dynamicize the rpath).

If you put an absolute path in DT_RUNPATH you wouldn't be able to move the binary+library to a different directory. In other words you can't distribute it in a tarball. And you can't rely on the bindmount to put it in some arbitrary path because ld.so needs to find it for the binary outside the isolate, too. So you'd have to use both DT_RUNPATH with $ORIGIN and a bindmount. I'd hesitate to rely on it looking in /deps/ though. I think it got that because dirname "/memfd:isolate_memfd (deleted)" gives / but that doesn't seem like something to rely on. You can put multiple entries in DT_RUNPATH so I guess you could put both an $ORIGIN-relative path and an arbitrary absolute one for the bindmount.

did you try setting it in the Isolate::run call's env vars map?

OK, I just tried that, and it did not seem to work. Specifically I tried

let mut env = HashMap::new();
env.insert("LD_ORIGIN_PATH".to_owned(), "/home/cdruid/src/extrasafe_example/example/target/debug".to_owned());
let output = Isolate::run("isolate_print", &env).unwrap();

and got

[pid 32252] execve("/proc/self/fd/3", ["isolate_print"], ["LD_ORIGIN_PATH=/home/cdruid/src/extrasafe_example/example/target/debug"] <unfinished ...>
...
[pid 32252] <... execve resumed>)       = 0
[pid 32252] brk(NULL)                   = 0x56526e4d0000
[pid 32252] mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fde6d20d000
[pid 32252] readlinkat(AT_FDCWD, "/proc/self/exe", "/memfd:isolate_memfd (deleted)", 4096) = 30
[pid 32252] access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
[pid 32252] openat(AT_FDCWD, "//deps/glibc-hwcaps/x86-64-v3/libdylib.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 32252] newfstatat(AT_FDCWD, "//deps/glibc-hwcaps/x86-64-v3/", 0x7ffc2fcef3c0, 0) = -1 ENOENT (No such file or directory)
[pid 32252] openat(AT_FDCWD, "//deps/glibc-hwcaps/x86-64-v2/libdylib.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 32252] newfstatat(AT_FDCWD, "//deps/glibc-hwcaps/x86-64-v2/", 0x7ffc2fcef3c0, 0) = -1 ENOENT (No such file or directory)
[pid 32252] openat(AT_FDCWD, "//deps/libdylib.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 32252] newfstatat(AT_FDCWD, "//deps/", 0x7ffc2fcef3c0, 0) = -1 ENOENT (No such file or directory)

Which is the same behavior despite LD_ORIGIN_PATH being set in the execve syscall.

Apparently according to https://www.technovelty.org/linux/exploring-origin.html LD_ORIGIN_PATH is actually a fallback if readlink("/proc/self/exe") fails, unfortunately for us.

To be clear: for me this is an academic exercise; I simply got curious while reading your blog post and wanted to share what I learned. This bug isn't blocking me or anything like that.

[boustrophedon]

Apparently according to https://www.technovelty.org/linux/exploring-origin.html LD_ORIGIN_PATH is actually a fallback if readlink("/proc/self/exe") fails, unfortunately for us.

That is unfortunate. I think probably the best thing to do if someone were in this situation would just be to use patchelf to fix the library path to not be dynamic.

Anyway, thanks for reading my blog post and teaching me about $ORIGIN!