Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cosmopolitan Libc: build-once run-anywhere C library

justine.lol

Post reply on HN
Print view View original

11–20 of 171 posts

Re: Cosmopolitan Libc: build-once run-anywhere C library

#11

Post by jart »

qlk1123 wrote: post #4

Sorry for being naive, but how can it be possible that nobody had thought of this approach and this approach would have actually worked? What have been preventing people from coming up with something like this?

Author here. Here's the best explanation I've thought of so far:

> If it's this easy, why has no one done this before? The best answer I can tell is it requires an minor ABI change, where C preprocessor macros relating to system interfaces need to be symbolic. This is barely an issue, except in cases like switch(errno){case EINVAL:...}. https://justine.lol/ape.html

It also took an act of willpower. This is a novel approach to building software that says the conventional wisdom is wrong. When I first stumbled upon the idea, I couldn't believe it was even possible, until I actually did it. Now I'm glad that I did. Actually Portable Executable has been so useful for my work. I used to work on the TensorFlow team, so I've seen the worst c/c++ build toil imaginable, and looking back at those experiences I just can't believe how I managed to get by for so long without having portable build-once binaries without interpreters.

Re: Cosmopolitan Libc: build-once run-anywhere C library

#12

Post by jws »

Doesn’t the “strlcpy” example near the bottom of the page overwrite the buffer by one byte when the source string is as long or longer than the buffer?

Imagine I have a three byte buffer, n is 3 and I copy in “foo”. The result of the MIN is going to be three. And d[3] is the fourth byte of my three byte buffer.

Re: Cosmopolitan Libc: build-once run-anywhere C library

#13

Post by modeless »

mmastrac wrote: post #3

It appears to build a "naked" thompson shell script without a shebang line that works as a PE file for Windows/DOS, and uses `exec` to feed a binary to the shell via a pipe. Interesting approach for a universal binary.

A downside of this approach is the binary can't be executed with the exec system call because the kernel doesn't recognize scripts without a shebang. It can only be executed from within a shell.

Re: Cosmopolitan Libc: build-once run-anywhere C library

#14

Post by comex »

jws wrote: post #12

Doesn’t the “strlcpy” example near the bottom of the page overwrite the buffer by one byte when the source string is as long or longer than the buffer? Imagine I have a three byte buffer, n is 3 and I copy in “foo”. The result of the MIN is going to be three. And d[3] is the fourth byte of my three byte buffer.

Seems that way to me as well.

Re: Cosmopolitan Libc: build-once run-anywhere C library

#15

Post by harikb »

jws wrote: post #12

Doesn’t the “strlcpy” example near the bottom of the page overwrite the buffer by one byte when the source string is as long or longer than the buffer? Imagine I have a three byte buffer, n is 3 and I copy in “foo”. The result of the MIN is going to be three. And d[3] is the fourth byte of my three byte buffer.

I think you are right. `strlcpy` is only supposed to copy `n-1` bytes from src to dest.

https://github.com/freebsd/freebsd/blob/master/sys/libkern/s...

Re: Cosmopolitan Libc: build-once run-anywhere C library

#16

Post by jes »

jart wrote: post #11
qlk1123 wrote: post #4

Sorry for being naive, but how can it be possible that nobody had thought of this approach and this approach would have actually worked? What have been preventing people from coming up with something like this?

Author here. Here's the best explanation I've thought of so far: > If it's this easy, why has no one done this before? The best answer I can tell is it requires an minor ABI change, where C preprocessor macros relating to system interfaces need to be symbolic. This is barely an issue, except in cases like switch(errno){case EINVAL:...}. https://justine.lol/ape.html It also took an act of willpower. This is a novel ap…

Have you tried to code-sign executables produced by this method?

Re: Cosmopolitan Libc: build-once run-anywhere C library

#17

Post by viraptor »

rodneyzeng wrote: post #8

wow, most likely, this means there would be a virus infective to all operating systems...

It's not really necessary for that case. At the level of taking over control, you're attacking a specific system, so you need to know what it is and can provide specific payload. For the later phases, the dropper knows where it's running so it can request a specific version of the virus as well.

So there's nothing stopping multi-os viruses right now.

Re: Cosmopolitan Libc: build-once run-anywhere C library

#18

Post by jart »

jws wrote: post #12

Doesn’t the “strlcpy” example near the bottom of the page overwrite the buffer by one byte when the source string is as long or longer than the buffer? Imagine I have a three byte buffer, n is 3 and I copy in “foo”. The result of the MIN is going to be three. And d[3] is the fourth byte of my three byte buffer.

Author here. The example and repository has been updated. Many eyes make all bugs shallow. I was hoping to do more testing before posting on Hacker News. Now that the cat's out of the bag, I'd like to explain what's being done to make sure bugs like that never happen.

Cosmopolitan implements a simplified version of the Address Sanitizer (ASAN) runtime. See https://github.com/jart/cosmopolitan/blob/master/libc/log/as... I've been systematically working towards using it to vet the entire codebase. It's helped to address more than a few memory issues so far. However more work needs to be done polishing the ASAN code so it can be bootstrapped for the purposes of having memory safety in the lowest level libraries like LIBC_STR, due to the way that the Makefile requires acyclicity between libraries.

This has been less of an issue, because bugs are normally very evident in functions like strcpy(). The BSD version is an exception though, since it was only used to port programs from the OpenBSD codebase which were used sparingly. But in either case, the real focus here is having continual incremental improvements over time, and I'm very committed to addressing any and all concerns.

Cosmopolitan also has extensive unit tests. This is something that C libraries have traditional done somewhat sparingly, if at all, since C library authors conventionally rely on an ecosystem of ported software that very quickly breaks if everything isn't perfect. If you build the Cosmopolitan repository, you'll notice there's 194 independent test executables in the o/test folder. That's one of the great things about Actually Portable Executable. It makes binaries so tiny that we can have lots of tests, completely isolated from each other!

Re: Cosmopolitan Libc: build-once run-anywhere C library

#19

Post by jart »

Sebb767 wrote: post #7
qlk1123 wrote: post #4

Sorry for being naive, but how can it be possible that nobody had thought of this approach and this approach would have actually worked? What have been preventing people from coming up with something like this?

I can think of two reasons: It's arguably a hack and it's not needed. While this is awesome, it can easily break (zsh already seems to do so) and, as the author states, it is not meant for GUI apps or programs relying deeply on OS APIs, which includes a lot of apps. Additionaly, it does not solve an important problem. Doing a build for each supported OS is not that hard and easily automated. The actual problem is to…

Author here. I intend to upstream a patch soon with zsh that restores backwards compatibility with the Thompson Shell.

As for GUIs, while Cosmpolitan isn't intended to write complex GUI apps (since I normally build web apps for that) it still can be used to build WIN32 GUIs. See https://justine.lol/apelife/index.html as an example of Conway's Game of Life built with Cosmopolitan, where it manages to embed a UNIX TUI and a WIN32 GUI in the same binary! Even if you turn the WIN32 off, it'll still run the TUI inside the Windows Command Prompt.

As for breakages, Cosmopolitan only depends on stable kernel ABIs with long term stability promises. The binaries you build will work on all Linux distros dating back to RHEL5, Windows versions dating back to Vista, etc. There's no reason to believe there will be any radical breaking changes to these ABIs. Why is it so stable? Alignment of self-interest. By using canonical kernel interfaces, we ensure platforms can't break us without first breaking themselves.

Post reply on HN