No. 07 of 09

Heron500M

Half a billion parameters, comfortable on any modern laptop.

Versionv1

Specifications

Architecture
Decoder-only transformer
Parameters
495.5M
Layers
31
Hidden size
1,152
Attention heads
18 query · 6 key/value
Head dimension
64
Feed-forward size
3,072
Context length
4,096 tokens
Vocabulary
49,152
Attention
Grouped-query attention
QK norm
Per-head RMSNorm
Positions
RoPE, θ = 50,000
Activation
SwiGLU
Normalization
RMSNorm, pre-norm
Embeddings
Tied input / output
Languages
French, English
Training budget
49.8B tokens
License
TBA

Planned configuration for v1. Final numbers are confirmed at release. See Architecture.

Heron 500M Base Planned

The raw pre-trained model. It continues text rather than following instructions. Use it as the starting point for your own fine-tuning.

Get the weights

  • Aulvero/Heron-500M-Base-v1Not yet released

Quick start

# pip install transformers torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Aulvero/Heron-500M-Base-v1"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

inputs = tok("La marée monte parce que", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=64)
print(tok.decode(out[0], skip_special_tokens=True))

This snippet will work once Heron Base is published.

Heron 500M Instruct Planned

Fine-tuned from Base to follow instructions and hold a conversation in French or English. Most people should start here.

Get the weights

  • Aulvero/Heron-500M-Instruct-v1Not yet released

Quick start

# pip install transformers torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Aulvero/Heron-500M-Instruct-v1"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

messages = [{"role": "user", "content": "Explique les marées en deux phrases."}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
out = model.generate(inputs, max_new_tokens=128)
print(tok.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))

This snippet will work once Heron Instruct is published.

Footprint

991 MBBF16 weights
273 MB4-bit weights
MediumLaptops and single GPUs