Plan & Roadmap
The roadmap follows the scikit-learn mapping faithfully, respecting dependencies between crates. Development methodology is as important as the destination. The wave decomposition below comes from the wave-plan-foundation planning change (OpenSpec) — every downstream change references it as the shared reference.
Every change starts with a specification (proposal, deltas, design, tasks) on a dedicated -openspec branch, reviewed before any code.
Test first, confirmed failure, minimal implementation, refactoring — with companion *_tests.rs modules and mock data in ndarray/sprs.
No direct commits to main. Every change lives in a git worktree with small, focused PRs.
An independent agent validates that the spec was effectively met before every PR.
Iterative evolution per algorithm
No stage may be skipped — each algorithm walks the full path:
Acceptance checklist (every implementation)
- Runs correctly with lots of data and with little data
- Correct under concurrency (automatic + applicable explicit modes)
- Exports model (minimum Safetensors) and produces metrics
- Companion
_tests.rsmodules covering everything above - Complete names, no abbreviations (single exception:
sk/SK) - Public items with mandatory prefix; methods unprefixed
- No file over 200 lines without folder-module organization
- Python binding in an associated change; GPU backends in separate changes
Phases
Workspace bootstrap + CI · sciencekit_common traits + central errors · sciencekit_math kernel (pure-Rust faer default) · automatic execution decision + builders + tracing/OTel + allocators.
SKStandardScaler, SKMinMaxScaler, SKRobustScaler (tdigest online quantile sketch), SKOneHotEncoder, SKPolynomialFeatures; imputation strategies (SKSimpleImputer). SKKNNImputer moved to Wave 3, built directly on the trees — no brute-force placeholder.
Linear/Ridge/Lasso/ElasticNet/Logistic regressions; SGD classifier/regressor with streaming (SKLazySource).
k-NN classifier/regressor, KD-tree, Ball-tree; SVC, SVR, LinearSVC.
Decision trees; Random Forest, Bagging, Gradient Boosting aggregated in parallel with rayon.
KMeans, MiniBatchKMeans (streaming), DBSCAN; PCA, TruncatedSVD; anomaly detection.
KFold, train/test split, GridSearchCV; type-safe SKPipeline with DAGs; full metrics suite.
Safetensors sharding/padding/compression, ONNX export/import, Polars/DataFusion sources; umbrella crate; complete PyO3 bindings.
Wave plan and foundational decisions
The wave-plan-foundation planning change decomposes PRD phases 0–7 into waves W0–W7 — roughly 60 downstream changes — so dependencies are respected naturally and no throwaway/placeholder implementations are needed. It also locks six foundational technical decisions that every downstream change references:
faer 0.24.4 is the default (SKFaerBackend); matrixmultiply is the GEMM fallback; ndarray-linalg+blas-src stays an opt-in blas-backend. oxiblas was disqualified: it does not compile on MSRV 1.85.
OpenCL 3.0 first, ICD-agnostic via opencl3 + the OpenCL loader (vendor-transparent); Metal after; CUDA/ROCm only on request. Heavy algebra stays on CPU.
tdigest 1.0.0 powers SKRobustScaler::partial_fit — a capability scikit-learn lacks (percentiles require the full column).
faer-sparse + rsvd-faer for SKTruncatedSVD; hand-rolled Lanczos fallback; arpack-sys as opt-in. single-svdlib (gold-standard IRLBA) waits on an MSRV bump.
rayon owns all parallelism (work-stealing caps threads); BLAS runs single-threaded inside rayon scopes. No semaphore, no threadpoolctl-style hack.
SKKNNImputer lives in Wave 3, built directly on SKKDTree/SKBallTree — no brute-force placeholder or searcher-trait hack.
Deferred tracker (ADR #37): the nalgebra role for small fixed-size matrices was deferred at W0.3 — not adopted, since sciencekit_math has no small-matrix hot paths yet. Revisit at W5.4 (PCA), adopting nalgebra 0.35 only if faer’s small-matrix performance proves inadequate for 2×2/3×3 covariance.