Models

AuXSpec-9B-RL and AuXSpec-27B-RL generate and repair Lean specifications and implementations. Each model covers four tasks: SpecGen, CodeGen, SpecGen-Repair, and CodeGen-Repair.

Example run

The model reads a problem description, its constraints, and its test cases, and writes a Lean specification.

Problem description

Perfect Squares. Given an integer n, return the least number of perfect square numbers that sum to n.

Constraints

  • 1 <= n <= 10^4

Test cases

Examples

  • n = 12 → 3
  • n = 13 → 2
  • n = 8 → 2

Invalid inputs

  • n = 0
  • n = 10001
  • n = 100000

Wrong outputs

  • n = 12 → 4
  • n = 13 → 4
  • n = 8 → 5
∀AuXSpec modelDone
perfect-squares.lean99 lines
Imports and typesimport Mathlib abbrev Input := Natabbrev Output := NatSpecification helpers/-! ### Specification helpers -/abbrev IsPerfectSquare (k : Nat) : Prop := ∃ m : Nat, m * m = kValidity predicate/-! ### Validity predicate (V) -/def isValidInput (input : Input) : Bool :=1 ≤ input && input ≤ 10000Declarative postcondition/-! ### Ideal specification (P_spec) -/def numSquares_spec (input : Input) (result : Output) : Prop :=∃ l : List Nat, l.sum = input ∧ (∀ x ∈ l, IsPerfectSquare x) ∧l.length = result ∧∀ t : List Nat, t.sum = input →(∀ x ∈ t, IsPerfectSquare x) → result ≤ t.lengthExecutable postcondition/-! ### Decidable specification (P_check) -/def squareAlphabet (input : Nat) : List Nat :=(List.range (Nat.sqrt input + 1)).map (fun j => j * j) def listsOver (alphabet : List Nat) : Nat → List (List Nat)| 0 => [[]]| L' + 1 =>(listsOver alphabet L').flatMap fun l => alphabet.map fun a => l ++ [a] def sumMatches (input : Nat) (l : List Nat) : Bool :=l.sum == input def numSquares_check (input : Input) (result : Output) : Bool :=let alphabet := squareAlphabet inputlet existsAt (L : Nat) : Bool :=(listsOver alphabet L).any fun l => sumMatches input lexistsAt result && (List.range result).all fun L => !existsAt LEquivalence theorem/-! ### Equivalence theorem -/theorem numSquares_check_iff_spec (input : Input) (result : Output)(h : isValidInput input) :numSquares_check input result ↔ numSquares_spec input result := bysorryTest cases/-! ## Unit tests -//-! ### L1 — Validity guard (V accepts the provided examples) -/#guard isValidInput 12#guard isValidInput 13#guard isValidInput 8 /-! ### L1b — Negative validity (V rejects invalid input) -/#guard !(isValidInput 0)#guard !(isValidInput 10001)#guard !(isValidInput 100000) /-! ### L2 — Specification guard (P_check accepts the correct output) -/#guard numSquares_check 12 3#guard numSquares_check 13 2#guard numSquares_check 8 2 /-! ### L2b — Negative specification (P_check rejects a wrong output) -/#guard !(numSquares_check 12 4)#guard !(numSquares_check 13 4)#guard !(numSquares_check 8 5)
  • CompilesChecked by Lean 4
  • 12/12 test cases passRun by Lean 4
  • Judges approveChecked by LLM judges

Figure 1. Recorded output on Perfect Squares from AuXSpec-Test, abridged and replayed faster than real time.

Try AuXSpec-27B-RL

If the model has been idle, the first run takes about 3 minutes to start.

Weights and usage

ModelBackboneContext
AuXSpec-9B-RLQwen3.5-9B32,768 tokens
AuXSpec-27B-RLQwen3.8-27B32,768 tokens

Both models are served with vLLM in BF16 and answer through the chat template of the backbone, with thinking enabled. Each task has its own system prompt, and the user message holds the problem description, the constraints, and the test cases.

The evaluation draws 4 samples per problem at temperature 0.3 and top-p 0.95, with up to 16,384 output tokens.

Request body for /v1/chat/completions
{
  "model": "auxspec-9b-rl",
  "messages": [
    {"role": "system", "content": "<system prompt of the task>"},
    {"role": "user", "content": "<problem description, constraints, and test cases>"}
  ],
  "temperature": 0.3, "top_p": 0.95, "top_k": 20, "presence_penalty": 1.0,
  "max_tokens": 16384,
  "chat_template_kwargs": {"enable_thinking": true}
}

Full results

RL improves SpecGen pass@1 on AuXSpec-Test from 10.1 to 58.6 (9B) and from 18.7 to 65.7 (27B). Both RL models also transfer to Verina, an out-of-domain benchmark.

ModelAuXSpec-TestVerina
SpecGenCodeGenSpecGen-RepairCodeGen-RepairSpecGenCodeGen
pass@1pass@4pass@1pass@4pass@1pass@4pass@1pass@4pass@1pass@4pass@1pass@4
Frontier models
Claude-Opus-557.781.393.098.383.090.186.995.186.295.256.679.4
Claude-Sonnet-513.839.354.873.066.481.777.288.364.890.517.941.8
GPT-5.6-Sol29.859.780.096.361.877.182.389.385.197.920.840.7
GPT-5.6-Luna9.528.739.971.349.469.571.887.467.394.724.558.2
Kimi-K343.883.767.294.373.587.881.397.177.696.334.972.5
DeepSeek-V4-Pro21.254.348.672.357.682.476.994.253.882.537.670.4
DeepSeek-V4-Flash38.476.745.069.767.088.573.395.160.292.65.818.5
Instruct models
Qwen3.5-9B0.20.78.218.08.426.04.914.67.018.523.449.2
Qwen3.8-27B4.514.38.922.044.773.335.764.128.855.03.313.2
Prior work
VeriSpecGen-4B†––––––––30.345.235.466.8
VeriSpecGen-30B-A3B†––––––––47.663.958.784.0
AuXSpec
AuXSpec-9B-SFT10.124.313.128.356.776.359.580.637.369.838.070.4
AuXSpec-9B-RL58.682.777.196.073.584.774.594.287.096.892.698.4
AuXSpec-27B-SFT18.740.727.652.773.184.772.387.457.382.557.886.8
AuXSpec-27B-RL65.784.394.899.781.989.388.195.191.996.891.893.7
Table 1. Pass@1 and pass@4 (%) on AuXSpec-Test (300 problems) and Verina (189 problems, test cases in the prompt). Bold marks the best value in each column and underline the second best. † Reported in the original paper without test cases in the prompt, and left out of the comparison. Source: paper, Tables 4 and 5.