Mini-LLM - Transformer Language Model in Rust

The Challenge

Deeply understanding how large language models work internally by rebuilding a GPT-style Transformer from scratch, without relying on the usual Python ecosystem. The challenge was to reimplement every mathematical building block (multi-head attention, residual connections, layer normalization, loss computation and backpropagation) in a statically typed systems language, while keeping the code readable and educational rather than over-optimized.

The Solution

Complete implementation in Rust with Candle, Hugging Face's ML framework, structured into clear modules, the tokenizer (character-level), the model (Transformer architecture) and the training loop. Every component (token and positional embeddings, attention blocks, FFN, output head) is written explicitly and commented. Text generation is autoregressive with temperature-controlled sampling. It all compiles into a single binary exposing a CLI to train and generate, with optional GPU support (CUDA / Metal).

Results