Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fedora 38 LLVM vs. Team Fortress 2

airlied.blogspot.com

Post reply on HN
Print view View original

101–110 of 111 posts

Re: Fedora 38 LLVM vs. Team Fortress 2

#101

Post by bravetraveler »

0x457 wrote: post #71
bravetraveler wrote: post #39

Earlier quoted context omitted.

I was thinking/wondering this myself. Not to reinvent the wheel - more toss an idea around, but a 'venv for LD_PRELOAD' sounds like it'd deal with this pretty handily Not... in a way I'd use as a distribution/release maintainer. Probably as an administrator [of my LAN]

That's Nix with extra steps.

I specifically said I'm not really trying to solution this, lol. More toying with the LD_PRELOAD aspect than anything

Nix is neat, and I don't think I've used it enough to be too critical - but in some ways it feels like 'extra steps'

I wanted to make a 'reproducible' installation (ala kickstart, not strictly binary)... but it felt very much like distribution work; declaring dependencies and the like

Re: Fedora 38 LLVM vs. Team Fortress 2

#102

Post by admax88qqq »

gabcoh wrote: post #35
admax88qqq wrote: post #32

Unfortunately this is exactly the type of stuff that makes supporting commercial apps on linux a nightmare. Weird crashes due to weird linking of system libraries. Common distros are very adamant about dynamic linking everything in order to support the use case of "core library has vulnerability, upgrade it in place without rebuilding consuming apps." Along with a desire to avoid "dll hell" and force a single canonic…

Can Linux not trivially do the same thing as windows with LD_PRELOAD? If so why is this more of an issue on Linux than Windows? Is it really less a technical challenge and more just a matter of Linux getting less support from upstream developers?

Yes linux _can_, the machinery is there, but culturally the common distros do not. And the defaults do not. On windows I can literally drop a DLL next to an executable and it will pick it up. On linux I have to do a wrapper script to set LD_PRELOAD, or mutate the binary's rpath to get it to load.

It's not really a question of capability, but a question of culture and defaults that makes linux hard to support.

Debian for example goes through great pains (or used to at least) to unbundle shared libraries such as openssl from projects like chromium.

Re: Fedora 38 LLVM vs. Team Fortress 2

#103

Post by never_inline »

bravetraveler wrote: post #39
gabcoh wrote: post #35

Earlier quoted context omitted.

Can Linux not trivially do the same thing as windows with LD_PRELOAD? If so why is this more of an issue on Linux than Windows? Is it really less a technical challenge and more just a matter of Linux getting less support from upstream developers?

I was thinking/wondering this myself. Not to reinvent the wheel - more toss an idea around, but a 'venv for LD_PRELOAD' sounds like it'd deal with this pretty handily Not... in a way I'd use as a distribution/release maintainer. Probably as an administrator [of my LAN]

There are tools which overwrite linked libraries, eg: chrpath.

Re: Fedora 38 LLVM vs. Team Fortress 2

#104

Post by olliej »

DannyBee wrote: post #50
olliej wrote: post #4

This is a predictable outcome of overriding the global operator new. It remains annoying that this was ever allowed, and is a constant source of pain for c++ standard library implementations.

It actually should still work, since fedora38 includes the llvm15 versioned libs. The only way to make this break is if something is loading random unversioned solibs or whatever the latest one it can find is, and expecting this to work forever. If it actually used a versioned solib, it would get llvm 15 just like it did before. This is the whole point of versioned solibs.

Versioning does not solve the problem.

The aligned allocation operators have existed since llvm 8.x.

The problem is not that the aligned allocation APIs are new. The problem is that TCMalloc is only partially replacing the global allocation APIs, it's just taken until this year for that bug to be exposed.

What has happened is presumably some part of the OS has updated its target C++ version so is now using the aligned allocators, which exposes the gap in TCMalloc.

I'm not sure if the spec explicitly allows an aligned allocation to be fed into an unaligned operator delete, but it seems like implementations do, so that's probably why adopting aligned operator new wouldn't be seen as an ABI break.

Re: Fedora 38 LLVM vs. Team Fortress 2

#105

Post by 0x457 »

bravetraveler wrote: post #101
0x457 wrote: post #71

Earlier quoted context omitted.

That's Nix with extra steps.

I specifically said I'm not really trying to solution this, lol. More toying with the LD_PRELOAD aspect than anything Nix is neat, and I don't think I've used it enough to be too critical - but in some ways it feels like 'extra steps' I wanted to make a 'reproducible' installation (ala kickstart, not strictly binary) ... but it felt very much like distribution work; declaring dependencies and the like

Oh, nix is an extra mile. A lot could be improved, but that's what I'm using to deal with dependencies.

Re: Fedora 38 LLVM vs. Team Fortress 2

#106

Post by boutique »

kimixa wrote: post #77
boutique wrote: post #76

Funnily enough, on Half-Life 1 engine-based games (i.e. the engine that came before HL2 - on which Team Fortress 2 runs; such as Counter-Strike 1.6), a different allocator problem exists -- glibc's malloc() just decides to fail miserably[0] on some setups. [0] https://github.com/ValveSoftware/halflife/issues/3158

that's exactly the sort of error you get if something has written just out of bounds on a malloc'd chunk - it clobbers the allocator's internal state, which appears to be what that assert() is checking. It's probably an allocation before the failing one that is being misued - so the backtrace pointing to openal doesn't necessarily mean it's openal's fault. Running with valgrind or another heap memory checking tool wi…

Thanks a lot for the guidance/tip, I've learned something new. And you're absolutely right about the cause of the mentioned crash -- I've updated the Github issue with a bit of new info I've gathered.

Regarding the function, here it is: https://github.com/dreamstalker/rehlds/blob/master/rehlds/fi...

Interestingly, strdup gets compiled into:

  89 04 24           mov   [esp+101Ch+name], pszContentPath ; s
  E8 82 DC 00 00     call  strlen
  66 C7 04 03 2F 00  mov   word ptr [pszContentPath+eax], 2Fh ; '/'
Which is basically:

  *(_WORD *)&pPath[strlen(pPath)] = '/';`
and would explain why Valgrind says it goes one byte over.

Re: Fedora 38 LLVM vs. Team Fortress 2

#107

Post by bravetraveler »

0x457 wrote: post #105
bravetraveler wrote: post #101

Earlier quoted context omitted.

I specifically said I'm not really trying to solution this, lol. More toying with the LD_PRELOAD aspect than anything Nix is neat, and I don't think I've used it enough to be too critical - but in some ways it feels like 'extra steps' I wanted to make a 'reproducible' installation (ala kickstart, not strictly binary) ... but it felt very much like distribution work; declaring dependencies and the like

Oh, nix is an extra mile. A lot could be improved, but that's what I'm using to deal with dependencies.

Gotcha, I don't feel so floundering now!

I plan to spend more time with it, I see a lot of merit

The amount of control is great, but the docs could use some work. For my simple goals (install Sway, Ansible, some other things) it was a broadsword when I need a butter knife

Re: Fedora 38 LLVM vs. Team Fortress 2

#108

Post by kimixa »

boutique wrote: post #106
kimixa wrote: post #77

Earlier quoted context omitted.

that's exactly the sort of error you get if something has written just out of bounds on a malloc'd chunk - it clobbers the allocator's internal state, which appears to be what that assert() is checking. It's probably an allocation before the failing one that is being misued - so the backtrace pointing to openal doesn't necessarily mean it's openal's fault. Running with valgrind or another heap memory checking tool wi…

Thanks a lot for the guidance/tip, I've learned something new. And you're absolutely right about the cause of the mentioned crash -- I've updated the Github issue with a bit of new info I've gathered. Regarding the function, here it is: https://github.com/dreamstalker/rehlds/blob/master/rehlds/fi... Interestingly, strdup gets compiled into: 89 04 24 mov [esp+101Ch+name], pszContentPath ; s E8 82 DC 00 00 call strlen…

Yeah, looks like the Q_strcat(pszContentPath, "/"); is invalid, as glibc has only allocated exactly enough to fit the path in the buffer returned by realpath().

The compiler seems to completely inline the strcat and write the '/' and null as a single 2-byte word write, the null then being out of bounds of the malloc'd chunk and likely causing the error as it overwrites something important.

Interestingly, the open group spec says that a null argument to realpath is "Implementation defined" [0]

And the linux (glibc) man pages say it allocates a buffer "Up to PATH_MAX" [1]

I guess "strlen(path)" is "Up to PATH_MAX", but the man page seems unclear - you could read that as implying the buffer is always allocated to PATH_MAX size, but that's not what seems to be happening, just effectively calling strdup() [2]. I have no idea how to feed back to the linux man pages, but might be worth clarifying there.

[0] https://pubs.opengroup.org/onlinepubs/009696799/functions/re...

[1] https://linux.die.net/man/3/realpath

[2] https://github.com/bminor/glibc/blob/0b9d2d4a76508fdcbd9f421...

Re: Fedora 38 LLVM vs. Team Fortress 2

#109

Post by 0x457 »

bravetraveler wrote: post #107
0x457 wrote: post #105

Earlier quoted context omitted.

Oh, nix is an extra mile. A lot could be improved, but that's what I'm using to deal with dependencies.

Gotcha, I don't feel so floundering now! I plan to spend more time with it, I see a lot of merit The amount of control is great, but the docs could use some work. For my simple goals (install Sway, Ansible, some other things) it was a broadsword when I need a butter knife

What sold me on nix is home-manager and flakes: I can easily bootstrap my environment anywhere nix is available.

Re: Fedora 38 LLVM vs. Team Fortress 2

#110

Post by olliej »

nemetroid wrote: post #97
olliej wrote: post #61

Earlier quoted context omitted.

Haha, you've missed the issue. The question is what does the system do when someone overrides the builtin allocator functions, but does not override all of them. You are absolutely correct that as a developer you can have your process override the allocator functions, and that is in fact what TF has done. The problem is that they have not overridden all of the allocation functions, and so they're crashing due to mism…

> Haha, you've missed the issue. That's not very nice. The root comment said nothing about making the system handle this automatically, it just described an idea for a potential fix to be applied to this particular case: > It should be straightforward to make a little LD_PRELOAD shim to implement the new operator new on top of old overloads and thus restore proper functioning.

> That's not very nice.

:(

It was not intended as a dismissive or derisive laugh at the author, but a laugh at the absurdity of the issue itself. Think "haha, you'd think that the reason is X, but technology is involved, and so everythong is terrible" vs "haha you're dumb" which sure as heck was not my intended message.

Post reply on HN