Large Language Models for Compiler Optimization
1–10 of 119 posts
Re: Large Language Models for Compiler Optimization
#2This kind of application of LLMs is most interesting to me, since it's possible to evaluate correctness and performance quantitatively.
Re: Large Language Models for Compiler Optimization
#3This kind of application of LLMs is most interesting to me, since it's possible to evaluate correctness and performance quantitatively.
It seems like a poor fit to me precisely because correctness is boolean, difficult to measure and getting it wrong is very bad. I do think there's a place for AI here but it's probably not LLMs in their current form.
Re: Large Language Models for Compiler Optimization
#4There's a half day tutorial at the LLVM Developers Meeting on this, ML-Guided Compiler Optimization in LLVM. However, the authors of this paper aren't giving that tutorial.
Re: Large Language Models for Compiler Optimization
#5> understanding.
We evaluate on a large suite of test programs. Our approach achieves a 3.0% improvement in reducing instruction counts over the compiler,
3% code size reduction is really good. The challenge will be having codegen like this that someone is willing to support. And for that they'd want to be able to reason about why the compiler made this decision or that one. IIUC that's an outstanding problem for AI in general.
Re: Large Language Models for Compiler Optimization
#6This kind of application of LLMs is most interesting to me, since it's possible to evaluate correctness and performance quantitatively.
It seems like a poor fit to me precisely because correctness is boolean, difficult to measure and getting it wrong is very bad. I do think there's a place for AI here but it's probably not LLMs in their current form.
The approach seems to focus on selecting optimizations to apply for LLVM (e.g., imagime: should this be inlined as an example), but the worst case is that the code is poorly optimized, you can't select optimizations to apply and get a wrong result.
I agree that what you say is true of compilation as a whole, but that doesn't seem to be the focus here (rather, it's used as a sort of crutch to help the LLM learn)
Re: Large Language Models for Compiler Optimization
#7This kind of application of LLMs is most interesting to me, since it's possible to evaluate correctness and performance quantitatively.
It seems like a poor fit to me precisely because correctness is boolean, difficult to measure and getting it wrong is very bad. I do think there's a place for AI here but it's probably not LLMs in their current form.
They are not using LLM to directly produce the result code, but as tool that lists which optimisations should be done and in which order, which is fairly complex problem to solve. But if optimisation passes are implemented correctly (which is anyway required for a functioning optimising compiler), it cannot produce incorrect code, maybe only suboptimal compared to default heuristics used.
Re: Large Language Models for Compiler Optimization
#8This kind of application of LLMs is most interesting to me, since it's possible to evaluate correctness and performance quantitatively.
It seems like a poor fit to me precisely because correctness is boolean, difficult to measure and getting it wrong is very bad. I do think there's a place for AI here but it's probably not LLMs in their current form.
The trick is finding a way to ensure that LLM produces something which is always correct. Like in this case, the LLM only changes compiler optimizations, not the assembly itself, so no matter what it outputs the code is correct, it just may be larger. Other possibilities: an LLM which applies semantics-preserving program transformations, or an LLM combined with a proof assistant to verify the output (more generally and for any domain, an LLM as an NP oracle).
But I agree, as of now I haven't seen good uses where LLMs produce reliable output. Not only do you need that guarantee that whatever output always generates a correct program, you need something where an LLM is considerably better than a simple or random algorithm, and you need a lot of training data (severely restricting how creative you can be with the output).
Re: Large Language Models for Compiler Optimization
#9There's a half day tutorial at the LLVM Developers Meeting on this, ML-Guided Compiler Optimization in LLVM . However, the authors of this paper aren't giving that tutorial.
MLGO uses RL
Re: Large Language Models for Compiler Optimization
#10This kind of application of LLMs is most interesting to me, since it's possible to evaluate correctness and performance quantitatively.
It seems like a poor fit to me precisely because correctness is boolean, difficult to measure and getting it wrong is very bad. I do think there's a place for AI here but it's probably not LLMs in their current form.
Quantification can be done by measuring in at least two dimensions: (1) the size of the synthesised code, and (2) how precisely the generated code matches the input (which means roughly: on what fraction of input do the two programs give different output). We have set up a challenge that seeks to entice the community to look into this problem domain more. And we've simplified the assumptions, so as to make it more tractable:
- Challenge: https://codalab.lisn.upsaclay.fr/competitions/15096
- Paper describing the challenge: https://arxiv.org/abs/2308.07899
(I am one of the authors, AMA)