[flagged]
Re: source code producing different binaries: things like ASLR, stack canaries, optimization levels, linking, etc all lead to different binaries.
I hate compilers
21–30 of 182 posts
Re: I hate compilers
#22LLMs should be trained on and directly output binary.
Generative algorithms have been studied for decades now and while they have led to some interesting results they're a bad fit for LLMs because there's no such thing as a "plausible" binary: a small perturbation yields an unusable result.
Re: I hate compilers
#23If Clang generated non-deterministic output due to pointer addresses then that's a bug (happens regularly) that should be fixed. The most common way this happens if it some code path is iterating over a DenseMap which is non-deterministic. Sometimes that's fine and sometimes that's not depending on how that map is used. The common way to fix that is to switch to a MapVector which pays some additional runtime/memory c…
I'll try and make a minimal reproduction case and file a bug. Do you know if any tooling that can take a binary and fuzz it down to a minimal reproduction set?
Re: I hate compilers
#24Earlier quoted context omitted.
On the off chance that you’re serious, that would result in disastrously bad output. The difference between “jmp $+15” and “jmp $+16” is inscrutable and the LLM would not be able to pick the right one without tooling. That tooling is a compiler. The higher level, the better chance the LLM can be steered to good output. Machine code is hopeless, don’t bother.
What about AOT optimization? whuch brings aot closer to JITs performance? Isn't that something LLM + Harness can easily do?
I think the idea that AOT is inherently faster than JIT, or vice versa, is a thoroughly debunked idea.
You can have LLMs help you optimize code but I don’t think you can do this unattended for non-trivial code.
Re: I hate compilers
#25Time date env variables and random address... Is also input data, maybe not as a flag but still
Re: I hate compilers
#26Earlier quoted context omitted.
> The difference between “jmp $+15” and “jmp $+16” is inscrutable I don't see why that's the case. LLM trained on binary would totally see it, not? Also the tool can also be running the test and a debugger.
> I don't see why that's the case. LLM trained on binary would totally see it, not? It would not. You find the correct version by counting the number of bytes to the destination. LLMs are famously bad at this kind of problem (counting). > Also the tool can also be running the test and a debugger. The test needs to provide a good amount of signal. That’s too hard if you are throwing machine code at the wall. In order…
Can "LLMs are bad at counting" be generalized to "LLM are better in complex stuff but make more mistakes in simple"?
Re: I hate compilers
#27LLMs should be trained on and directly output binary.
Technically they are, just a subset. But still a practical one, they're frequently used to produce executable files.
Re: I hate compilers
#28Time date env variables and random address... Is also input data, maybe not as a flag but still
Time and date are... tolerable. There's SOURCE_DATE_EPOCH which should always be set to whack it into submission when used. ASLR of the _compiler being invoked_ resulting in a difference in the _program being compiled_ is nuts and would break any self-hosting compiler with consistency checks.
Re: I hate compilers
#29Earlier quoted context omitted.
> I don't see why that's the case. LLM trained on binary would totally see it, not? It would not. You find the correct version by counting the number of bytes to the destination. LLMs are famously bad at this kind of problem (counting). > Also the tool can also be running the test and a debugger. The test needs to provide a good amount of signal. That’s too hard if you are throwing machine code at the wall. In order…
Can "LLMs are bad at counting" be generalized to "LLM are better in complex stuff but make more mistakes in simple"?
No, I don’t think so. LLMs are good at a lot of simple tasks, but bad at certain simple tasks. Moravec’s paradox in a new iteration.
It applies to humans too. Calculus is “simple” but it takes something like sixteen years to train a human to do it, if all goes well. Meanwhile, most humans think that inverse kinematics is, like, the easiest thing in the world (it’s a super complicated task).
Re: I hate compilers
#30Earlier quoted context omitted.
> I don't see why that's the case. LLM trained on binary would totally see it, not? It would not. You find the correct version by counting the number of bytes to the destination. LLMs are famously bad at this kind of problem (counting). > Also the tool can also be running the test and a debugger. The test needs to provide a good amount of signal. That’s too hard if you are throwing machine code at the wall. In order…
Can "LLMs are bad at counting" be generalized to "LLM are better in complex stuff but make more mistakes in simple"?
Its more LLMs are better at vague problems with multiple non perfect solutions, and struggle at problems that require precision.