Large Language Models Surprise Meta AI Researchers at improved Compiler Optimization!

Good news! This is only the beginning!

"Meta AI Researchers were trying to make Large Language Models (LLMs) do the same kind of code optimizations that regular compilers, like LLVM, do. LLVM’s optimizer is incredibly complex, with thousands of rules and algorithms written in over 1 million lines of code in the C++ programming language. ...

Their approach is straightforward, starting with a 7-billion-parameter Large Language Model (LLM) architecture sourced from LLaMa 2 and initializing it from scratch. The model is then trained on a vast dataset consisting of millions of LLVM assembly examples, each paired with the best compiler options determined through a search process for each assembly, as well as the resulting assembly code after applying those optimizations. Through these examples alone, the model acquires the ability to optimize code with remarkable precision.

The notable contribution of their work lies in being the first to apply LLMs to the task of code optimization. They create LLMs specifically tailored for compiler optimization, demonstrating that these models achieve a 3.0% improvement in code size reduction on a single compilation compared to a search-based approach that attains 5.0% improvement with 2.5 billion compilations. In contrast, state-of-the-art machine learning approaches lead to regressions and require thousands of compilations. ..."

Large Language Models Surprise Meta AI Researchers at Compiler Optimization! - MarkTechPost

Large Language Models for Compiler Optimization