LLMs as Copilots for Theorem Proving in Lean

Lean Copilot: LLMs as Copilots for Theorem Proving in Lean

Lean Copilot allows large language models (LLMs) to be used in Lean for proof automation, e.g., suggesting tactics/premises and searching for proofs. You can use our built-in models from LeanDojo or bring your own models that run either locally (w/ or w/o GPUs) or on the cloud.

Table of Contents

1. Requirements
2. Using Lean Copilot in Your Project
   1. Adding Lean Copilot as a Dependency
   2. Getting Started with Lean Copilot
      1. Tactic Suggestion
      2. Proof Search
      3. Premise Selection
3. Advanced Usage
   1. Tactic APIs
   2. Model APIs
   3. Bring Your Own Model
4. Caveats
5. Getting in Touch
6. Acknowledgements
7. Citation

Requirements

• Supported platforms: Linux, macOS and Windows WSL
• Git LFS
• Optional (recommended if you have a CUDA-enabled GPU): CUDA and cuDNN
• Required for building Lean Copilot itself (rather than a downstream package): CMake >= 3.7 and a C++17 compatible compiler

Using Lean Copilot in Your Project

⚠️ Your project must use a Lean version of at least lean4:v4.3.0-rc2.

Adding Lean Copilot as a Dependency

1. Add the package configuration option moreLinkArgs := #["-L./.lake/packages/LeanCopilot/.lake/build/lib", "-lctranslate2"] to lakefile.lean. Also add the following line:

require LeanCopilot from git "https://github.com/lean-dojo/LeanCopilot.git" @ "v1.0.0"

3. Run lake update LeanCopilot
4. Run lake exe LeanCopilot/download to download the built-in models from Hugging Face to ~/.cache/lean_copilot/
5. Run lake build

Here is an example of a Lean package depending on Lean Copilot. If you have problems building the project, our Dockerfile, build.sh or build_example.sh may be helpful.

Getting Started with Lean Copilot

Tactic Suggestion

After import LeanCopilot, you can use the tactic suggest_tactics to generate tactic suggestions. You can click on any of the suggested tactics to use it in the proof.

You can provide a prefix (e.g., simp) to constrain the generated tactics:

Proof Search

The tactic search_proof combines LLM-generated tactics with aesop to search for multi-tactic proofs. When a proof is found, you can click on it to insert it into the editor.

Premise Selection

The select_premises tactic retrieves a list of potentially useful premises. Currently, it uses the retriever in LeanDojo to select premises from a fixed snapshot of Lean and mathlib4.

Running LLMs

You can also run the inference of any LLMs in Lean, which can be used to build customized proof automation or other LLM-based applications (not limited to theorem proving). It's possible to run arbitrary models either locally or remotely (see Bring Your Own Model).

Advanced Usage

This section is only for advanced users who would like to change the default behavior of suggest_tactics, search_proof, or select_premises, e.g., to use different models or hyperparameters.

Tactic APIs

• Examples in TacticSuggestion.lean showcase how to configure suggest_tactics, e.g., to use different models or generate different numbers of tactics.
• Examples in ProofSearch.lean showcase how to configure the proof search using options provided by aesop.
• Examples in PremiseSelection.lean showcase how to set the number of retrieved premises for select_premises.

Model APIs

Examples in ModelAPIs.lean showcase how to run the inference of different models and configure their parameters (temperature, beam size, etc.).

Lean Copilot supports two kinds of models: generators and encoders. Generators must implement the TextToText interface:

class TextToText (τ : Type) where
  generate (model : τ) (input : String) (targetPrefix : String) : IO $ Array (String × Float)

• input is the input string
• targetPrefix is used to constrain the generator's output. "" means no constraint.
• generate should return an array of String × Float. Each String is an output from the model, and Float is the corresponding score.

We provide three types of Generators:

• NativeGenerator runs locally powered by CTranslate2 and is linked to Lean using Foreign Function Interface (FFI).
• ExternalGenerator is hosted either locally or remotely. See Bring Your Own Model for details.
• GenericGenerator can be anything that implements the generate function in the TextToText typeclass.

Encoders must implement TextToVec:

class TextToVec (τ : Type) where
  encode : τ → String → IO FloatArray

• input is the input string
• encode should return a vector embedding produced by the model.

Similar to generators, we have NativeEncoder, ExternalEncoder, and GenericEncoder.

Bring Your Own Model

In principle, it is possible to run any model using Lean Copilot through ExternalGenerator or ExternalEncoder (examples in ModelAPIs.lean). To use a model, you need to wrap it properly to expose the APIs in external_model_api.yaml. As an example, we provide a Python API server and use it to run a few models, including llmstep-mathlib4-pythia2.8b.

Caveats

• Lean may occasionally crash when restarting or editing a file. Restarting the file again should fix the problem.
• select_premises always retrieves the original form of a premise. For example, Nat.add_left_comm is a result of the theorem below. In this case, select_premises retrieves Nat.mul_left_comm instead of Nat.add_left_comm.

@[to_additive]
theorem mul_left_comm : ∀ a b c : G, a * (b * c) = b * (a * c)

• In some cases, search_proof produces an erroneous proof with error messages like fail to show termination for .... A temporary workaround is changing the theorem's name before applying search_proof. You can change it back after search_proof completes.

Getting in Touch

• For general questions and discussions, please use GitHub Discussions.
• To report a potential bug, please open an issue. In the issue, please include your OS information and the exact steps to reproduce the error. The more details you provide, the better we will be able to help you.
• Feature requests and other suggestions are extremely welcome. Please feel free to start a discussion!

Acknowledgements

• We use the frontend of llmstep for displaying tactics.
• We thank Scott Morrison for suggestions on simplifying Lean Copilot's installation and Mac Malone for helping implement it. Both Scott and Mac work for the Lean FRO.
• We thank Jannis Limperg for supporting our LLM-generated tactics in Aesop (leanprover-community/aesop#70).

Citation

Below is our workshop paper describing an earlier version of Lean Copilot. We plan to release the full paper in early 2024. Stay tuned!

@inproceedings{song2023towards,
  title={Towards Large Language Models as Copilots for Theorem Proving in {Lean}},
  author={Song, Peiyang and Yang, Kaiyu and Anandkumar, Anima},
  comment={The last two authors advised equally.},
  booktitle={The 3rd Workshop on Mathematical Reasoning and AI at NeurIPS'23},
  year={2023}
}