Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Configuration file (TOML)

Reference for the optional --configuration-file accepted by train. Values resolve with the precedence CLI flag > TOML key > default. Every key is optional; a partial file is valid.

For a guided version, see Configuring a run.

[run]

TOML keyTypeCLI flagDefault
methodstring--methodlora
seedinteger--seed42
output_directorystring--output-directoryoutput/train
model_idstring--model-idQwen/Qwen2.5-1.5B-Instruct
quantizationstring--quantizationnone
quantization_modestring--quantization-modepost-training
devicestring--deviceauto
maximum_gradient_normfloat--maximum-gradient-norm1.0
minimum_improvementfloat--minimum-improvement0.0
early_stop_patienceinteger--early-stop-patience0
max_sequence_lengthinteger--max-sequence-length1024
warmup_stepsinteger--warmup-steps10
weight_decayfloat--weight-decay0.0
learning_ratefloat--learning-rate1e-4
batch_sizeinteger--batch-size4
gradient_accumulation_stepsinteger--gradient-accumulation-steps1
epochsinteger--epochs3
lora_rankinteger--lora-rank16
lora_alphafloat--lora-alpha32.0
lora_dropoutfloat--lora-dropout0.0

[model]

TOML keyTypeCLI flag
architecturestring--architecture
vocab_sizeinteger--vocab-size
hidden_sizeinteger--hidden-size
intermediate_sizeinteger--intermediate-size
num_hidden_layersinteger--num-hidden-layers
num_attention_headsinteger--num-attention-heads
head_diminteger--head-dim
num_key_value_headsinteger--num-key-value-heads
max_position_embeddingsinteger--max-position-embeddings
rope_thetafloat--rope-theta
rms_norm_epsfloat--rms-norm-eps
tie_word_embeddingsboolean--tie-word-embeddings
attention_biasboolean--attention-bias
sliding_windowinteger--sliding-window
sliding_window_patterninteger--sliding-window-pattern
rope_local_base_frequencyfloat--rope-local-base-frequency
query_pre_attention_scalarinteger--query-pre-attention-scalar
logit_softcappingfloat--logit-softcapping
attention_logit_softcappingfloat--attention-logit-softcapping

[initialization]

No CLI flag; applied over the defaults for from-scratch.

TOML keyTypeDefaultMeaning
initializer_rangefloat0.02Std dev of attention/MLP projections
embedding_stdfloat0.02Std dev of token embeddings (and untied head)
norm_weightfloat1.0Constant for every RMSNorm weight
bias_valuefloat0.0Constant for attention-projection biases

[dataset]

TOML keyTypeCLI flagDefault
pathstring--dataset— (required)

[tokenizer]

TOML keyTypeCLI flagDefault
filestring--tokenizer-file— (required by from-scratch)

Full example

[run]
method = "from-scratch"
seed = 42
output_directory = "output/scratch"
model_id = "Qwen/Qwen2.5-1.5B-Instruct"
quantization = "none"
quantization_mode = "post-training"
device = "auto"
maximum_gradient_norm = 1.0
minimum_improvement = 0.0
early_stop_patience = 0
max_sequence_length = 1024
warmup_steps = 10
weight_decay = 0.0
learning_rate = 1e-4
batch_size = 4
gradient_accumulation_steps = 1
epochs = 3
lora_rank = 16
lora_alpha = 32.0
lora_dropout = 0.0

[model]
architecture = "qwen3"
vocab_size = 151936
hidden_size = 1024
intermediate_size = 4096
num_hidden_layers = 16
num_attention_heads = 16
num_key_value_heads = 4
max_position_embeddings = 4096
rope_theta = 1000000.0
rms_norm_eps = 1e-6
tie_word_embeddings = true

[initialization]
initializer_range = 0.02
embedding_std = 0.02
norm_weight = 1.0
bias_value = 0.0

[dataset]
path = "resources/dataset.jsonl"

[tokenizer]
file = "tokenizer.json"

Errors

Unknown keys and wrong types are rejected and name the file and key:

configuration file error in 'training.toml': unknown field `epocs`, expected one of ...
configuration file error in 'training.toml': invalid type: string "three", expected usize ...