Quick start

This guide takes you from zero to a typed answer, then trains and serves a LoRA adapter. The fastest path uses the published container images — no toolchain to install; the cargo path follows for those who prefer a native binary. No GPU is required for the first steps.

1. Install

With Docker (recommended)

The images are published to the GitHub Container Registry on every release, tagged latest and with the version:

docker pull ghcr.io/neurono-ml/typed-lm-serve:latest
docker pull ghcr.io/neurono-ml/typed-lm-trainer:latest

With cargo

# Server and trainer from crates.io (CPU build).
cargo install typed-lm-serve
cargo install typed-lm-trainer

# Or build from the workspace.
cargo build --release --workspace

For a CUDA build, add --features cuda; on Apple Silicon, add --features metal. See Running the server for the full matrix.

2. Start the server

With Docker

The server downloads the default model on first startup and listens on 8080. Pass an HF_TOKEN for gated models and mount a context file to anchor the answers on your own facts:

docker run --rm -p 8080:8080 \
  -e HF_TOKEN=<hugging-face-token> \
  -v "$PWD/resources/memory.md:/etc/typed-lm/memory.md:ro" \
  -e CONTEXT_PATH=/etc/typed-lm/memory.md \
  ghcr.io/neurono-ml/typed-lm-serve:0.1.1

To persist the downloaded weights across runs, add a volume for the Hugging Face cache:

docker run --rm -p 8080:8080 \
  -e HF_TOKEN=<hugging-face-token> \
  -v typed-lm-cache:/root/.cache/huggingface \
  -v "$PWD/resources/memory.md:/etc/typed-lm/memory.md:ro" \
  -e CONTEXT_PATH=/etc/typed-lm/memory.md \
  ghcr.io/neurono-ml/typed-lm-serve:0.1.1

With cargo

# Downloads the default model (Qwen/Qwen2.5-1.5B-Instruct) on first startup.
typed-lm-serve --context-path resources/memory.md

Verify it is up (works for both paths):

curl -s http://127.0.0.1:8080/health/live
curl -s http://127.0.0.1:8080/health
curl -s http://127.0.0.1:8080/v1/models

The server listens on 0.0.0.0:8080 by default.

3. Ask your first questions

curl -s http://127.0.0.1:8080/v1/systemone \
  -H 'Content-Type: application/json' \
  -d @examples/request_mixed.json

You receive typed answers: a probability for noul, a winning label with a distribution for choice, and an expected value with a legend for score.

{
  "model": "typed-lm",
  "answers": {
    "refund_eligible": { "type": "noul", "noul": 0.87 },
    "responsible_department": {
      "type": "choice",
      "choice": "logistics",
      "probabilities": { "billing": 0.05, "logistics": 0.9, "product_support": 0.05 },
      "confidence": 0.85
    }
  },
  "usage": { "input_tokens": 512, "output_tokens": 4 }
}

The complete contract is in Calling the API.

4. Train a LoRA adapter

With Docker

Mount the working directory so the checkpoint, the dataset and the output all live on the host. The container runs with /work as the working directory:

docker run --rm -v "$PWD:/work" -w /work \
  -e HF_TOKEN=<hugging-face-token> \
  ghcr.io/neurono-ml/typed-lm-trainer:0.1.1 train \
  --model-id /work/checkpoint \
  --dataset /work/resources/dataset.jsonl \
  --output-directory /work/output/train \
  --method lora --epochs 3 --batch-size 4 --learning-rate 1e-4

For GPU training, use the :cuda image and add --gpus all (the host needs the NVIDIA driver and the container toolkit):

docker run --rm --gpus all -v "$PWD:/work" -w /work \
  -e HF_TOKEN=<hugging-face-token> \
  ghcr.io/neurono-ml/typed-lm-trainer:cuda train \
  --model-id /work/checkpoint --dataset /work/resources/dataset.jsonl \
  --output-directory /work/output/train --method lora --device cuda

With cargo

typed-lm-trainer train \
  --model-id /path/to/local/checkpoint \
  --dataset resources/dataset.jsonl \
  --output-directory output/train \
  --method lora --epochs 3 --batch-size 4 --learning-rate 1e-4

The dataset format and every flag are documented in Preparing datasets and Training LoRA and QLoRA adapters.

5. Quantize and serve

With Docker

docker run --rm -v "$PWD:/work" -w /work \
  -e HF_TOKEN=<hugging-face-token> \
  ghcr.io/neurono-ml/typed-lm-trainer:0.1.1 quantize \
  --model-id /work/checkpoint \
  --adapter-directory /work/output/train \
  --quantization fp8 --output-directory /work/output/quantized

# The quantized directory holds weights only; add the base metadata.
cp /path/to/local/checkpoint/config.json    output/quantized/
cp /path/to/local/checkpoint/tokenizer.json output/quantized/

# Serve the artifact.
docker run --rm -p 8080:8080 \
  -e HF_TOKEN=<hugging-face-token> \
  -v "$PWD/output/quantized:/models/quantized:ro" \
  ghcr.io/neurono-ml/typed-lm-serve:0.1.1 \
  --model-id /models/quantized

On a GPU, use the :cuda image and add --gpus all:

docker run --rm --gpus all -p 8080:8080 \
  -e HF_TOKEN=<hugging-face-token> \
  -v "$PWD/output/quantized:/models/quantized:ro" \
  ghcr.io/neurono-ml/typed-lm-serve:cuda \
  --model-id /models/quantized

With cargo

typed-lm-trainer quantize \
  --model-id /path/to/local/checkpoint \
  --adapter-directory output/train \
  --quantization fp8 --output-directory output/quantized

# The quantized directory holds weights only; add the base metadata.
cp /path/to/local/checkpoint/config.json    output/quantized/
cp /path/to/local/checkpoint/tokenizer.json output/quantized/

typed-lm-serve --model-id output/quantized

See Quantization (FP8 and FP4) and Serving a trained artifact.

Where to go next

• System One decisions — the mental model.
• Questions (primitives) — noul, choice and score.
• Training overview — the full training tutorial.
• HTTP API reference — every route and field.