No. 07 of 09
Heron500M
Half a billion parameters, comfortable on any modern laptop.
Specifications
- Architecture
- Decoder-only transformer
- Parameters
- 495.5M
- Layers
- 31
- Hidden size
- 1,152
- Attention heads
- 18 query · 6 key/value
- Head dimension
- 64
- Feed-forward size
- 3,072
- Context length
- 4,096 tokens
- Vocabulary
- 49,152
- Attention
- Grouped-query attention
- QK norm
- Per-head RMSNorm
- Positions
- RoPE, θ = 50,000
- Activation
- SwiGLU
- Normalization
- RMSNorm, pre-norm
- Embeddings
- Tied input / output
- Languages
- French, English
- Training budget
- 49.8B tokens
- License
- TBA
Planned configuration for v1. Final numbers are confirmed at release. See Architecture.
Heron 500M Base Planned
The raw pre-trained model. It continues text rather than following instructions. Use it as the starting point for your own fine-tuning.
Get the weights
- Aulvero/Heron-500M-Base-v1Not yet released
Quick start
# pip install transformers torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "Aulvero/Heron-500M-Base-v1"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
inputs = tok("La marée monte parce que", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=64)
print(tok.decode(out[0], skip_special_tokens=True))This snippet will work once Heron Base is published.
Heron 500M Instruct Planned
Fine-tuned from Base to follow instructions and hold a conversation in French or English. Most people should start here.
Get the weights
- Aulvero/Heron-500M-Instruct-v1Not yet released
Quick start
# pip install transformers torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "Aulvero/Heron-500M-Instruct-v1"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
messages = [{"role": "user", "content": "Explique les marées en deux phrases."}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
out = model.generate(inputs, max_new_tokens=128)
print(tok.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))This snippet will work once Heron Instruct is published.
Footprint
991 MBBF16 weights
273 MB4-bit weights
MediumLaptops and single GPUs