Project Vision

The idea A complete, fast and safe ML library — keeping everything that makes scikit-learn loved.

Goal

sciencekit is a Machine Learning library written natively in Rust exposing every algorithm and utility of scikit-learn, rewritten from scratch in an optimized form. It is not a rushed port: each algorithm evolves through mandatory stages — naive implementation, tests, performance tuning (SIMD, parallelism, memory layout) and out-of-core support.

🚀

Extreme performance

Native code with autovectorization, explicit SIMD on profiled hot paths, data parallelism via rayon, and custom allocators (jemalloc/mimalloc) with an expected 15–30% gain in allocation throughput.

🔐

Memory safety

All of Rust's guarantees without a garbage collector. Models travel as Safetensors — an auditable format that eliminates pickle's attack surface.

🌊

Native out-of-core

Two complementary traits cover any dataset: SKLazySource for sequential streaming in batches and SKMappableSource for O(1) random access via memory-mapping.

⚙️

First-class concurrency

Computation never blocks async threads: rayon runs the math, Tokio handles I/O. The execution mode is user-configurable — or decided automatically from available RAM, dataset size and access pattern.

Positioning

Equivalent to scikit-learn in algorithmic coverage, with the performance and safety of Rust. Three target audiences:

Audience	Need served
ML engineers in Rust	A complete, idiomatic toolkit without rewriting algorithms
Python teams	High-performance inference/training via zero-copy PyO3 bindings
Production systems	Removing the pickle risk + models exportable to ONNX/Safetensors

Scope

Is: a Rust library in a workspace with sub-crates; an idiomatic native API with the builder pattern; Python bindings (PyO3) with zero-copy; complete coverage of the scikit-learn taxonomy; ONNX, Safetensors and JSON debug export/import.

Is not (for now): a CLI for running workflows (recorded as a future improvement); a deep learning framework; an inference server.

API principles

Mandatory builder pattern — every estimator, transformer and pipeline exposes a builder; direct constructors stay private. Every builder accepts execution_mode(SKExecutionMode::...), defaulting to Automatic.
Zero-copy inputs — public APIs receive ArrayView, CowArray or sparse views (sprs), never arrays by value.
Type-safe pipelines — step compatibility validated at compile time through associated types.
Full descriptive names — no abbreviations; the single exception is the project prefix sk/SK. Examples: maximum_number_of_iterations (not max_iter), nearest_neighbors_count (not k).

// Core trait vocabulary — implemented by the sciencekit crates.
pub trait SKEstimator { /* hyperparameters and fit */ }
pub trait SKRegressorPredictor: SKEstimator { /* predict -> Array1<F> */ }
pub trait SKClassifierPredictor: SKEstimator { /* labels -> Array1<i64> */ }
pub trait SKTransformer: SKEstimator { /* transform / fit_transform */ }

pub trait SKDataSource { /* eager: full in-memory access */ }
pub trait SKLazySource { /* streaming: Iterator<Item = Batch> */ }
pub trait SKMappableSource { /* memmap: O(1) random access */ }

Algorithms compose exactly the traits they support: an SKSGDClassifier implements SKClassifierPredictor + SKLazySource; an SKStandardScaler implements only SKTransformer.

Interoperability

Single internal format: extended Safetensors — JSON header with hyperparameters, training state and recoverability metadata (checkpointing), with partial writes via sharding or header padding.
ONNX: every estimator implements SKToOnnx; external models load as a generic predictor.
Data: pluggable sources behind arbitrary I/O traits, with conversions for Polars and DataFusion.
Observability: tracing oriented to OpenTelemetry, disableable at near-zero cost.

Source of truth: this site mirrors the PRD (docs/PRD.md). Relevant scope changes go through that document first, before becoming technical specs.

Next: the algorithms → Architecture