FrankenSciPy Rewriting the Numerical Library in Rust

01 · What it is

What SciPy is, and why it matters

SciPy is the open-source Python library for scientific and technical computing, built on NumPy. It is the mathematical backbone of scientific Python: optimization, integration and differential equations, interpolation, linear algebra, statistics and probability distributions, signal and image processing, sparse matrices, spatial data structures. If pandas is how data gets wrangled, SciPy is how it gets computed.

It matters for the least glamorous reason in software: trust. Switching costs for a numerical stack are dominated by the ecosystem, not the algorithms. SciPy's moat is NumPy/pandas/scikit-learn integration, decades of edge-case hardening, and the accumulated confidence of every downstream package's test suite. A library like this is infrastructure you bet your results on. Rewriting it means re-earning that trust, routine by routine.

02 · Why build it

Why rewrite it in Rust?

The maintainer states the problem in the README, and it is sharper than "Python is slow." SciPy's runtime is showing its age: the CPython GIL and Python's object model make it awkward to use as a high-throughput library inside a service, a notebook kernel cluster, or a multi-agent system. The numerical kernels themselves are fast wrapped C, Fortran, and LAPACK. But the Python glue layer is where memory churn, type-coercion overhead, and packaging headaches actually live. Numerical-stability decisions are buried inside per-routine heuristics, with no first-class way to ask "which algorithm did you pick for this problem, and how confident are you that it's the right one?" And embedding SciPy in a Rust application, a WebAssembly module, or a memory-constrained edge runtime is a non-starter.

The proposed solution carries three guarantees the original cannot provide: memory- and thread-safety by construction (#![forbid(unsafe_code)] workspace-wide); explicit conditioning-aware algorithm selection: every solve goes through a runtime portfolio that records the chosen action, the evidence that drove the choice, and the expected loss versus the alternatives; and differential conformance against the real SciPy, with Python oracle scripts capturing reference outputs inside automated test runs.

The engineering case: the synchronous, forbid(unsafe), FFI-free kernels are exactly the shape a browser or edge numerical substrate wants, and the decision machinery is new: a portfolio that picks the solver and emits an audit certificate proving the choice was justified. The assessment's inference [Inference, Medium] is that no incumbent numerical library ships runtime algorithm selection with an audit trail.

The counter-argument, which the packet keeps attached: SciPy's moat is its ecosystem, and FrankenSciPy cannot touch it. The README itself says "Can I use this from Python? Not directly today. There is no PyO3 layer." The stated audience (Rust, embedded, agent numerical work) is a niche that existing Rust crates already serve. A from-scratch rewrite of a 1,300-symbol surface, to reach users who were never in the SciPy ecosystem to begin with, is an expensive way to sell them the audit certificates.

03 · What was built

What the project actually built

Nineteen crates, 613,565 lines of first-party Rust: dense and sparse linear algebra, integration, interpolation, optimization, FFT, signal, spatial, special functions, statistics, clustering, ndimage, IO, constants, plus a runtime crate holding the portfolio engine and a conformance crate holding the differential harness. 10,174 inline tests, 795 conformance test files, 16 Python oracle scripts, 96 fuzz targets, 18 canonical parity packets. Zero unsafe blocks under a workspace-wide forbid. A real release: v0.2.0, git tag plus GitHub release, all 19 crates published on crates.io.

Three findings matter. First, CASP is a real idea. The Condition-Aware Solver Portfolio probes matrix conditioning, computes a posterior over condition states with a conformal calibrator, minimizes expected loss over solver actions against a calibrated loss matrix (the matrix values are verified in code, a real const fn), and returns a synchronous audit certificate. It is wired into five numerical domains, each entry point verified calling the selection routine. The calibration quality is maintainer-asserted, never independently validated. But the architecture is real.

Second, the scorecard disavows itself. The release scorecard file opens by telling you that not one figure in it clears the project's evidence gate. The FAQ refuses a speed headline: "For individual kernels, mileage varies by problem size, conditioning, and structure." For a "reimplementation of SciPy," the absence of quotable head-to-head numbers is a thesis-level gap, honestly disclosed.

Third, the parity census is exactly what it says it is, and the project says so. 1,300 of 1,300 routines are name-matched. The README admits the per-routine list of what each differential test actually asserts "does not exist yet" and that "referenced is weaker than compared." Parity here is a naming convention until the assertion census gets built.

04 · Where it stands

Where it stands

In our program's terms: TRL 5, NODUS ring Explore. In plain English: this is past lab validation. A real release on crates.io, green CI at the pin (the commit the assessment was pinned to), a working differential harness. But behavioral parity is census-level rather than assertion-level, downloads are trivial, and no independent party has validated any of it. It is substantive but unproven.

What would change the verdict, in order

05 · What it teaches

What this teaches about building with agents

Here is the transferable lesson, and it is about the instrument, not the code.

The repo's negative-evidence ledger records a finding about a single frankenlibc malloc/free primitive [Maintainer claim, Medium]: on the same worker, under two sanctioned harnesses, it measured 5.95× and 12.39×, and both A/A nulls passed. The fleet adopted the finding: harness disagreement is as large as worker disagreement, and a passing null does not certify the harness.

Anyone building with agents will run benchmarks, and the instinct is to control the worker: pin the machine, pin the binary, run A/A nulls. FrankenSciPy's lesson is that the harness is a variable too. source The measurement apparatus itself moves the ratio as much as the machine does. So: never aggregate numbers across harnesses, never quote a ratio without naming the harness that produced it, and when two sanctioned harnesses disagree with clean nulls, publish the disagreement as the finding. The teams that treat their benchmarks as instruments to be calibrated — rather than scoreboards to be filled — will be slower to claim victory and faster to actually earn it.