Start here

Introduction

What Aulvero is, how the model lineup is organised, and how to run the training code.

Aulvero trains compact bilingual language models for French and English, from 50 million to 900 million parameters. These docs cover how the models are designed, what data they learn from, and how to train one yourself with aulvero-train.

The lineup

Nine sizes share one architecture. Each size has a bird name, smallest to largest.

Model Params Tier Layers × width Context
Wren 50M Tiny 20 × 384 2,048
Finch 85M Tiny 21 × 512 2,048
Robin 125M Tiny 27 × 576 2,048
Lark 160M Small 29 × 640 4,096
Kestrel 250M Small 26 × 896 4,096
Osprey 350M Small 31 × 960 4,096
Heron 500M Medium 31 × 1,152 4,096
Crane 700M Medium 33 × 1,344 4,096
Condor 900M Medium 33 × 1,536 4,096

Each model ships in two variants:

  • Base is the raw pre-trained model, for your own fine-tuning.
  • Instruct is fine-tuned from Base to follow instructions and chat in French or English.

A Code variant (continued pre-training on code) is planned. See Naming for how names like Kestrel-250M-Instruct-v1 are built.

Repositories

Repository What it holds
aulvero-train Pre-training and fine-tuning code, model configs, data recipes
aulvero.github.io This website and these docs
huggingface.co/Aulvero Released weights

Quick start

You need Python 3.11+ and, for real runs, one or more NVIDIA GPUs. The tests run on a laptop CPU.

git clone https://github.com/aulvero/aulvero-train
cd aulvero-train
make install                 # .venv + dependencies
make test                    # unit tests, a few seconds on CPU

Check the pipeline end to end on the small pilot mix before spending real compute:

make prepare MIX=pilot       # download, tokenizer, packed token files
make pilot                   # 2,000-step run of a Wren-shaped model

Then train a real model:

make prepare                                 # main French / English / code mix
make pretrain MODEL=kestrel-250m GPUS=8      # -> Base
make finetune MODEL=kestrel-250m GPUS=8      # -> Instruct
make export   MODEL=kestrel-250m             # -> Hugging Face format

Run make help to list every command.

Repository layout

aulvero/          shared package: model, configs, data, chat format, LR schedules
pretrain/         pre-training entry point            -> Base models
finetune/         supervised fine-tuning entry point  -> Instruct models
configs/          one file per model + defaults.yaml shared by all of them
data/mixes/       pre-training data recipes (pretrain, code, pilot)
data/sft/         fine-tuning data (example.jsonl shows the format)
scripts/          data download, tokenizer, packing, launchers, export, sampling
tests/            unit and export tests

Generated files (data/raw, data/packed, artifacts, checkpoints, exports) are git-ignored. Weights belong on Hugging Face, not in git.