Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

System call instrumentation on Linux/x86‑64 using memory‑indirect calls, part I

humprog.org

Post reply on HN
Print view View original

11–20 of 34 posts

Re: System call instrumentation on Linux/x86‑64 using memory‑indirect calls, part I

#11

Post by mananaysiempre »

yjftsjthsd-h wrote: post #3
quotemstr wrote: post #2

Linux is unusual in OS kernels in that direct system calls from arbitrary userspace code are supported and ABI-stable. This model has always been a terrible idea. It robs the system of an ability to intercept system calls in userspace before doing an expensive privilege-mode transition. If, instead, as on OpenBSD, the kernel enforced the rule that all system calls had to go through libc (or perhaps a big ntdll.dll-li…

> This model has always been a terrible idea. It robs the system of an ability to intercept system calls in userspace before doing an expensive privilege-mode transition. This model has always been a trade-off. It has downsides, but it also has upsides, including an immense boost in flexibility; decoupling from any particular userspace is useful. > This way, you can just LD_PRELOAD in front of the VDSO and system cal…

> Can you LD_PRELOAD in front of the vDSO? I was under the (possibly mistaken) impression that the kernel injects it directly.

The kernel puts the vDSO in memory and tells ld.so where it is, but where if anywhere ld.so will put it in the search order it implements is its own concern. (TBH I don’t actually know whether ld.so will actually allow LD_PRELOAD to override the vDSO, but there’s no reason for it not to, except I guess for the syscalls that are needed to perform the dynamic linking itself.)

Re: System call instrumentation on Linux/x86‑64 using memory‑indirect calls, part I

#12

Post by razighter777 »

quotemstr wrote: post #2

Linux is unusual in OS kernels in that direct system calls from arbitrary userspace code are supported and ABI-stable. This model has always been a terrible idea. It robs the system of an ability to intercept system calls in userspace before doing an expensive privilege-mode transition. If, instead, as on OpenBSD, the kernel enforced the rule that all system calls had to go through libc (or perhaps a big ntdll.dll-li…

Direct system calls are an amazing idea. The NtDll and bsd models are worse. The whole libc becomes a security boundary without the protection of kernel space. So much windows malware and process tampering happens because now you have a library (ntdll) fully in userspace that is given special privileges, which now becomes a huge attack surface. Then you have to deal with breakages between the built in libc versions and the kernel

This syscall overhead isn't as much as you suppose it is; for workloads where the syscall overhead actually makes a difference there are robust low-syscall paths for io/latency sensitive operations with DPDK, io_uring, and futex being a few examples.

And there are robust performant methods on linux for syscall interception/tracing, see seccomp unotify, bpf tracepoints, ftrace.

Re: System call instrumentation on Linux/x86‑64 using memory‑indirect calls, part I

#13

Post by razighter777 »

freestanding wrote: post #7
quotemstr wrote: post #2

Linux is unusual in OS kernels in that direct system calls from arbitrary userspace code are supported and ABI-stable. This model has always been a terrible idea. It robs the system of an ability to intercept system calls in userspace before doing an expensive privilege-mode transition. If, instead, as on OpenBSD, the kernel enforced the rule that all system calls had to go through libc (or perhaps a big ntdll.dll-li…

[flagged]

yep and and it forces every application to deal with the C FFI. It's beautiful in linux that I can access the full kernel API from an int 0x80/syscall instruction + a few register loads without having to link against crap. I can write a simple cat utility in a dozen or so lines of assembly.

Re: System call instrumentation on Linux/x86‑64 using memory‑indirect calls, part I

#14

Post by PunchyHamster »

quotemstr wrote: post #2

Linux is unusual in OS kernels in that direct system calls from arbitrary userspace code are supported and ABI-stable. This model has always been a terrible idea. It robs the system of an ability to intercept system calls in userspace before doing an expensive privilege-mode transition. If, instead, as on OpenBSD, the kernel enforced the rule that all system calls had to go through libc (or perhaps a big ntdll.dll-li…

The amount of times we ran LD_PRELOAD in prod was vanishingly small and limited to debug so the OpenBSD solution seems to be just waste of CPU cycles

Re: System call instrumentation on Linux/x86‑64 using memory‑indirect calls, part I

#15

Post by orangesilk »

Joker_vD wrote: post #9
throwaway7356 wrote: post #6

Earlier quoted context omitted.

> all system calls had to go through libc (or perhaps a big ntdll.dll-like Which makes containers crap on Windows and *BSD as they have to run the currect libc or equivalent. Thus you need to build a different container per OS version which sucks compared to Linux.

Windows doesn't even have its own libc.

Windows does have three libc, likely as a compability layer. their names are:

  * 
  * msvcrt.dll, 2014
  * ucrt.dll (universal c runtime, since Windows 10)

Re: System call instrumentation on Linux/x86‑64 using memory‑indirect calls, part I

#16

Post by freestanding »

razighter777 wrote: post #13
freestanding wrote: post #7

Earlier quoted context omitted.

[flagged]

yep and and it forces every application to deal with the C FFI. It's beautiful in linux that I can access the full kernel API from an int 0x80/syscall instruction + a few register loads without having to link against crap. I can write a simple cat utility in a dozen or so lines of assembly.

[flagged]

Re: System call instrumentation on Linux/x86‑64 using memory‑indirect calls, part I

#17

Post by matheusmoreira »

quotemstr wrote: post #2

Linux is unusual in OS kernels in that direct system calls from arbitrary userspace code are supported and ABI-stable. This model has always been a terrible idea. It robs the system of an ability to intercept system calls in userspace before doing an expensive privilege-mode transition. If, instead, as on OpenBSD, the kernel enforced the rule that all system calls had to go through libc (or perhaps a big ntdll.dll-li…

> This model has always been a terrible idea.

I disagree. It's an amazing idea. It allows me to write freestanding programs without any C libraries. It allows compilers to have Linux system call builtins that directly generate the calling convention. I created an entire lisp interpreter with nothing but Linux system calls, completely freestanding.

I've written a sort of manifesto around this:

https://www.matheusmoreira.com/articles/linux-system-calls

> If I were Linus

Good thing you aren't.

As a kernel, Linux is completely independent from its user space. The instruction set is the correct abstraction for the system call entry point. There should be no "required C libraries". User space should be free to reinvent everything in Rust if it wants.

There are various kernel mechanisms for system call interception if that's what you want. Tools like strace work just fine on my lisp interpreter, so libc is clearly not needed.

LD_PRELOAD is a GNU ld feature. The linker is the exact sort of user space component that's supposed to be completely replaceable. None of this is any of Linux's business.

Use of the vDSO is not even mandatory. All system calls in the vDSO are also available via the kernel entry point. The vDSO is just an optimization for frequently called system calls like gettimeofday. Forcing all programs to use the vDSO would force them all to not only implement the ELF spec but also to implement a small ELF linker. This is a significant blow if you want to create minimal freestanding Linux programs.

Re: System call instrumentation on Linux/x86‑64 using memory‑indirect calls, part I

#18

Post by matheusmoreira »

yjftsjthsd-h wrote: post #3
quotemstr wrote: post #2

Linux is unusual in OS kernels in that direct system calls from arbitrary userspace code are supported and ABI-stable. This model has always been a terrible idea. It robs the system of an ability to intercept system calls in userspace before doing an expensive privilege-mode transition. If, instead, as on OpenBSD, the kernel enforced the rule that all system calls had to go through libc (or perhaps a big ntdll.dll-li…

> This model has always been a terrible idea. It robs the system of an ability to intercept system calls in userspace before doing an expensive privilege-mode transition. This model has always been a trade-off. It has downsides, but it also has upsides, including an immense boost in flexibility; decoupling from any particular userspace is useful. > This way, you can just LD_PRELOAD in front of the VDSO and system cal…

The vDSO is just a normal ELF shared object that Linux maps somewhere in the address space of the process. The kernel passes a pointer to the ELF header to the process via the auxiliary vector.

That's the end of Linux's involvement. It's up to the program itself to do something useful with that pointer, namely by parsing the ELF header, and then resolving its symbols to function pointer addresses.

There's no doubt that all the various libc implementations out there do this, but I don't know if they do it in a way that lets LD_PRELOAD override the vDSO. They could be hard linking the vDSO system calls into their system call stubs or something.

Usually programs intercept system calls by overriding the libc stubs, which also indirectly intercepts the vDSO. However, it's not actually a requirement that the system be structured like this. Theoretically, the program could do anything. System calls can be done directly, without any stubs. Compilers could just generate the code directly without any functions at all.

Re: System call instrumentation on Linux/x86‑64 using memory‑indirect calls, part I

#19

Post by matheusmoreira »

freestanding wrote: post #16
razighter777 wrote: post #13

Earlier quoted context omitted.

yep and and it forces every application to deal with the C FFI. It's beautiful in linux that I can access the full kernel API from an int 0x80/syscall instruction + a few register loads without having to link against crap. I can write a simple cat utility in a dozen or so lines of assembly.

[flagged]

You might enjoy my work on the lone lisp language. I got rid of the libc and implemented an entire interpreter with nothing but Linux system calls. Been working on it and blogging about it for about 3 years now.

http://github.com/lone-lang/lone/

Re: System call instrumentation on Linux/x86‑64 using memory‑indirect calls, part I

#20

Post by Joker_vD »

orangesilk wrote: post #15
Joker_vD wrote: post #9

Earlier quoted context omitted.

Windows doesn't even have its own libc.

Windows does have three libc, likely as a compability layer. their names are: * * msvcrt.dll, 2014 * ucrt.dll (universal c runtime, since Windows 10)

Those are not a compatibility layer with the OS. Heck, the all barely even provide proper access to the file system, ffs! The "msvcrt.dll" in the System32 folder is an ancient leftover from Microsoft-internal version of MSVC 6.0 or so, not intended for 3rd-party consumption.

At some point Microsoft got tired of maintaining binary-incompatible versions of its C runtime for different Visual Studios, so they started shipping UCRT with Windows itself... but you still don't need to touch that garbage for anything whatsoever.

Post reply on HN