Models
Architecture
The transformer design shared by every Aulvero model, the size table, and the tokenizer.
All Aulvero models share one architecture. Sizes differ only in depth, width, head count and feed-forward size, so one codebase and one export path cover the whole lineup.
Design
| Component | Choice | Why |
|---|---|---|
| Type | Dense decoder-only transformer | The standard for open small models (Llama 3, Qwen 3, SmolLM, Gemma 3) |
| Normalisation | RMSNorm, pre-norm | Stable and cheap |
| Attention | Grouped-query attention (GQA) | Smaller KV cache, so faster inference on phones and laptops |
| QK norm | RMSNorm on each query and key head | Stabilises training at higher learning rates (Qwen 3, OLMo 2, Gemma 3) |
| Positions | RoPE, θ = 50,000 | Leaves room to extend the context later |
| Feed-forward | SwiGLU | Better quality per parameter than GELU MLPs |
| Embeddings | Tied input and output | Saves 19–75M parameters, which matters most at small sizes |
| Biases | None | Simpler, with no quality loss |
| Head dimension | 64 at every size | Same kernels and export path everywhere |
Deep and thin
Below a billion parameters, models do better deep and narrow than shallow and wide at the same parameter count (MobileLLM, SmolLM2). Wren 50M has 20 layers and Condor 900M has 33, where a GPT-2-style layout would use 12–24.
Sizes
| Model | Params | Layers | Width | Q heads | KV heads | FFN | Context |
|---|---|---|---|---|---|---|---|
| wren-50m | 50.3M | 20 | 384 | 6 | 2 | 1,024 | 2,048 |
| finch-85m | 84.4M | 21 | 512 | 8 | 2 | 1,408 | 2,048 |
| robin-125m | 123.9M | 27 | 576 | 9 | 3 | 1,536 | 2,048 |
| lark-160m | 159.8M | 29 | 640 | 10 | 2 | 1,792 | 4,096 |
| kestrel-250m | 252.8M | 26 | 896 | 14 | 2 | 2,304 | 4,096 |
| osprey-350m | 352.0M | 31 | 960 | 15 | 5 | 2,560 | 4,096 |
| heron-500m | 495.5M | 31 | 1,152 | 18 | 6 | 3,072 | 4,096 |
| crane-700m | 702.0M | 33 | 1,344 | 21 | 7 | 3,584 | 4,096 |
| condor-900m | 906.1M | 33 | 1,536 | 24 | 8 | 4,096 | 4,096 |
make count prints this table from the configs. A test fails if any model drifts more than 3%
from the size in its name.
Tokenizer
One byte-level BPE tokenizer with 49,152 tokens, trained on the main French / English / code mix and shared by every size.
| Token | Use |
|---|---|
<bos> <eos> |
Document boundaries in pre-training |
<pad> <unk> |
Padding and unknown |
<|im_start|> <|im_end|> |
Chat turns (ChatML) for Instruct models |
<fim_prefix> <fim_middle> <fim_suffix> <file_sep> |
Reserved for the Code variant (fill-in-the-middle) |
Compatibility
Parameter names map one-to-one onto Hugging Face Qwen3ForCausalLM. scripts/export_hf.py converts
a checkpoint, and a test checks that the exported model produces the same outputs as the original.
Released models load with transformers, vLLM, llama.cpp and Ollama, with no Aulvero code needed.