Skip to contents

The FFI export surface (ffi-surface) review artifact

Source: vignettes/articles/ffi-surface.qmd

Status basis: vignettes/articles/architecture.qmd §2 (the FFI hard contracts 1–8), §3.5 (vignettes/articles/ffi-surface.qmd is the per-release review artifact); established at U-M0-09. Scope: every #[extendr] export of the root crate (the FFI shell) (msffi_*, registered into the msuiter dynamic library through the wrap__* of R/extendr-wrappers.R). The engine/catalog crates have no FFI surface and are not in this table. This file is a frozen artifact: the repository guard test (tests/testthat/test-ffi-surface-drift.R) parses the machine-readable manifest in §“the export manifest” and compares it item by item against the actual export set of the rextendr-generated wrappers; any drift (an addition/a removal/a renaming) turns the test red. Version-stamp semantics (frozen-at-M0, F-14 clarification, corpus pass 2026-10-13): every row of the detail table carries the single package-version stamp 0.0.0.9000 because the whole table was sealed as the frozen M0 baseline artifact (U-M0-09); the stamp therefore records the artifact’s freezing generation, not the per-export introduction version. The actual introduction of each later kernel export is recorded by the unit-ID reference inside its own row (U-M1s-09, U-M2-03, U-M3a-02, U-M3b-03, U-M4-01, U-M7-pre, …) and by the corresponding devlog entries. The stamped rows are never rewritten (see the convention below and frozen-protocol rule 2).

The export manifest (machine-readable; the drift guard treats it as the single source of truth) 

msffi_column_major_probe
msffi_na_probe
msffi_error_probe
msffi_interrupt_probe
msffi_thread_probe
msffi_nmf_replicates_probe
msffi_build_info
ms_tally_rust
ms_extract_rust
ms_pipeline_rust
ms_stratify_rust
ms_ard_rust
ms_sparse_rust
ms_fit_rust
ms_fit_bootstrap_rust
ms_test_presence_rust
ms_calibration_grid_rust
ms_match_solutions_rust
ms_compare_null_rust
ms_bootstrap_draws_rust
ms_compare_shape_null_rust

The export detail table

Export name Rust signature R wrapper Purpose Contract Status since
msffi_column_major_probe (x: R double matrix) -> f64 .msffi_column_major_probe(x) the column-major layout probe: returns the position-weighted check sum read in column-major order (the asymmetric golden round-trip; distinguishes “read by column” from “mistakenly read by row”) 2 diagnostic 0.0.0.9000
msffi_na_probe (x: R double matrix) -> i32 .msffi_na_probe(x) the NA/NaN second line of defense: after the R validator (anyNA, the first layer) the Rust side does an explicit scan + debug_assert; on discovery the whole call errors 3 diagnostic 0.0.0.9000
msffi_error_probe (i: i32, j: i32) -> f64 .msffi_error_probe(i, j) the end-to-end mapping of an out-of-bounds index → Err(MsError) → an R condition (class msuiter_error_rust, payload message/topic/i/j/c); the 1-based virtual 4x3 matrix 4, 5 diagnostic 0.0.0.9000
msffi_interrupt_probe (n_chunks: i32, n_threads: i32) -> i32 .msffi_interrupt_probe(n_chunks, threads) the interrupt-mechanism probe: the main thread pols R_CheckUserInterrupt at the chunk boundaries + the workers only see the AtomicBool + the per-call thread pool (n_threads resolved via .ms_resolve_threads(), 0 = the rayon default); returns the count of completed chunks 6, 7 diagnostic 0.0.0.9000
msffi_thread_probe (n_items: i32, seed: i32, n_threads: i32) -> Vec<f64> .msffi_thread_probe(n_items, seed, threads) the thread-invariance probe: the per-call ThreadPool, each item an independent PCG64 stream (the canonical layout v1), concatenated in the chunk-index order; the outputs for threads∈{1,N} are identical 6 diagnostic 0.0.0.9000
msffi_nmf_replicates_probe (counts: R double matrix, k: i32, replicates: i32, max_iter: i32, seed: i32, n_threads: i32) -> Vec<f64> .msffi_nmf_replicates_probe(counts, k, replicates, max_iter, seed, threads) the real-kernel parallel driver (U-M1s-05): runs replicates independent KL-NMF fits in parallel over the per-call ThreadPool (engine::nmf::fit_kl_on_stream), the seeded init of replicate r taken from StreamId{replicate: r, rank: 0, fold: 0} (the canonical layout v1); single-threaded within a unit (the kernel loop order untouched, A7), returning per replicate the final KL objective value in the replicate-index order – the outputs for threads∈{1,N} are identical; the interrupt = the main thread pols R_CheckUserInterrupt at the chunk boundaries + the workers only see the AtomicBool, either trigger errors the whole call with no partial result 6, 7 diagnostic 0.0.0.9000
msffi_build_info () -> List .msffi_build_info() the compile-time toolchain information (the rustc version / the target platform / the package version; recorded by build.rs), for the ms_sitrep() report 8 (build) diagnostic 0.0.0.9000
ms_tally_rust (genome_path: String, chrom: Vec<String>, pos: Vec<f64>, ref_: Vec<String>, alt: Vec<String>, sample: Vec<String>, strand: Vec<String>, want_sbs96: bool, want_sbs192: bool, want_sbs384: bool, want_sbs1536: bool, want_dbs78: bool, want_id83: bool, n_threads: i32) -> List .ms_tally_rust(genome_path, chrom, pos, ref_, alt, sample, strand, want_sbs96, …, threads) the catalog tally kernel (U-M1s-09, the first kernel export): one entry one exit – File::open + the read-only mmap of the 2bit reference genome (D12 the use-and-release; Mmap::map is the unsafe spot of the ffi shell); the records are stably sorted by (chrom,sample,pos) -> partitioned by (chromosome,sample) with each partition routed segment by segment through mnv::route_variants (the SPMG dinuc_sub==1 detection is within a single sample – coordinate-adjacent pairs across samples are never paired into a DBS and each is recorded as SBS, the U-M1s-09 audit fix; the split-VCF re-join / the DBS candidates / the skip ledger) -> the ±2 window / the 2×2 dinuc via the genome.rs context -> the REF-vs-genome and N checks (the SPMG parity: an N within ±2 removes from every SBS matrix; an N in the dinuc -> n_dinuc; a mismatch -> ref_mismatch; a window out of bounds -> context_bounds; all matches failing -> unknown_chrom) -> the counts SBS96/192/384/1536 + DBS78 (the channels×samples column-major flat buffer, the row order = the canonical order of channels.rs, the column order = the first-appearance order of the samples; a disabled table = empty; the DBS pairs are removed from every SBS matrix; SBS192 has no B/N channels and removes only from that table) -> the indel classification (the U-M1c ID83 wiring): every RoutedEvent::Indel is classified through indel83::assign_indel83_genome per the SPMG catalogue_generator_INDEL_single semantics (the anchor-to-genome alignment is at this layer): a channel exists -> the destination id83 (counted into the 83-channel matrix when want_id83 is on, the row order = the canonical ID83_CHANNELS order of indel83.rs; the interim skipped:simple_indel mapping of U-M1c-01 is retired); the N:Ins:M:x microhomology insertion (SPMG computes the key but does not write ID83, MMG:3738 iloc[:83]) -> the skip ins_microhomology_no_channel; the anchor base vs the reference genome mismatch (incl. an N anchor, SPMG :1408-1421) -> the skip indel_anchor_mismatch; the walk/MH window past the chromosome end (the structured hardening of the IndexError SPMG does not capture, D13) -> the skip context_bounds; the indel classification likewise runs independent of the switches (the ledger’s purity); the ledger is rendered along the mnv.rs SkipLedger (the input order, switch-independent); the partition-level parallelism (contracts 6/7, the U-M1s-11 follow-up): the (chromosome,sample) partitions are independent units chunked in parallel over the per-call ThreadPool (n_threads resolved via .ms_resolve_threads(), 0 = the rayon default sentinel; each worker holds its own 2bit parsing handle, single-threaded fixed record order within a partition; the single-column partition counts are merged in the partition order + the results are written back by the input index -> the thread-count invariance: the outputs for threads∈{1,4,16} are identical(), nailed by tests on both the Rust and the R side); a single partition or n_threads==1 degenerates to the sequential main-thread path; the interrupt = the main thread pols R_CheckUserInterrupt at the chunk (parallel) / partition (sequential) boundaries, the workers only read the AtomicBool (the partition start + every 4096 events), the sequential path keeps the main-thread poll granularity of every 4096 events; the 0 sentinel fixes the wave grid by the actual pool width (the audit P1 fix: workers=jobs.len() would collapse the wave count to 1 and degrade the interrupt to one check per call – the worst-case Ctrl-C latency ≈ the longest single partition (seconds for the largest human-genome chromosome), the partition count automatically grows finer with samples × chromosomes); interrupt: address-space held until process exit (frozen U-M0-09 limitation) – a user interrupt’s longjmp skips the Rust destructors, so this export’s read-only mmap address space is held until the process exits (backed by the page cache, the D12 statelessness unaffected); the C-trampoline hardening is deferred to M8/CI; the chrom exact match, no chr-prefix normalization (the M1s simplification, this row is the documentation declaration); returns List{ sbs96…dbs78 matrices, the id83 matrix (the registry labels land with the registry unit, for now only the column names), ledger, n_skipped, n_variants }. The FFI internal name: the user API is the ms_tally() generic (the U-M1s-11 assembly) 1, 2, 3, 4, 5, 7 kernel 0.0.0.9000
ms_extract_rust (counts: R double matrix = t(counts), n_samples×m_channels, k: i32, max_iter: i32, seed: i32, engine: String, n_threads: i32) -> List .ms_extract_rust(counts, k, max_iter, seed, engine, threads) the single-method extraction kernel (U-M1s-12): exactly one seeded NMF fit (the seeded init, the StreamId::ZERO canonical zero stream; the engine switch variants "kl" (β=1, the D6 default) / "eu" (β=2), anything else is an argument error; the replicates/consensus/K selection belong to M2 and are not in this export); the layout contract (the transpose trap) – the kernel eats the row-major m×n V, the R storage is column-major, hence the R side passes t(counts): the column-major flat buffer of an n×m matrix is exactly the row-major V (the correct identity: data[i*n+j] == counts[i,j], i=the 0-based channel, j=the 0-based sample – the audit correction: for the n×m column-major buffer of t(counts), the previously written data[j*m+i] does not hold), zero marshalling on the Rust side; the transpose guard = the asymmetric golden discipline (both the Rust extract.rs and the R-end smoke assert the reconstruction cosine ≥ 0.99 over a distinguishable ground truth at 96×12 (m≠n); passing the wrong direction must go red); the validation (contract 4): k∈[1, min(m,n)] (out of bounds -> argument, i=k, j=min(m,n)), max_iter/seed/n_threads non-negative, engine ∈ {kl, eu}, the kernel checks each cell finite/non-negative (the topic na/argument, the 1-based i/j); interrupt (contract 7, a documented decision): a single fit has no chunk structure – this export is declared bounded-duration and uninterruptible within max_iter (the alternative per-iteration poll would require copying the kernel loop into the ffi shell or adding a callback seam to the pure engine crate, both worse for M1s; the target scale 96×12×500 iterations ≪1s; the M2 replicates restore the poll through the existing chunked driver); n_threads is only validated ≥0 without building a pool (a single fit has nothing to parallelize, A7 trivially thread-invariant); returns the RAW unscaled factors List{ the signatures m×k column-major, the exposures k×n column-major, the objective of length max_iter+1, iterations } – the column normalization of W is a presentation convention of the assembly layer (R/extract.R) (scaling the rows of H by the same factor preserves the WH reconstruction), the kernel does not normalize. The FFI internal name: the user API is the ms_extract() generic (the U-M1s-12 assembly, MsSignature + the catalog_summary snapshot) 1, 2, 3, 4, 5 kernel 0.0.0.9000
ms_stratify_rust (counts: R double matrix column-major m×n (channels × samples, not transposed), manual_cutoff: f64, seed: f64, n_threads: i32) -> List .ms_stratify_rust(counts, manual_cutoff, seed, threads) (R/stratify.R, internal) the GMM hypermutator stratification kernel (the M2 planned slot executed early = the FFI wiring of U-M1c-02): the engine::stats::normalization_cutoff + classify_hypermutants semantics called with zero changes (the frozen SigProfiler rules: the k=2 GMM majority-cluster pruning -> trunc(mean+2σ), the manual_cutoff floor, the strictly-greater predicate; the MsRng StreamId::ZERO single stream across all the pruning iterations). The layout contract – the R side passes the m×n counts as-is (column-major, not transposed, the opposite of extract.rs’s t(counts)): the contiguous slice of column j is exactly sample j, and each sample’s totals = the in-column ascending per-channel f64 accumulation (exact for any accumulation order with integer-valued counts ⇒ byte-identical with R’s colSums); the transpose guard = the asymmetric golden discipline (the same cohort through the 1×n reference and the 4×n matrix (m≠1≠n) byte-equal, misreading the axis must go red, nailed on the Rust side by column_major_layout_guard_multichannel_matches_1row). The counts value domain (contract 4) – after the R wrapper’s first layer (matrix/REALSXP/anyNA), the Rust side scans cell by cell as the second layer: finite (the topic na) + non-negative + integer-valued (the topic argument, the payload the 1-based (i,j) column-major coordinates); n=0 -> argument; the buffer length ≠ m·n -> bounds. The wire format (the 0-based decision, this document is the declaration) – hypermutant_idx: Vec<i32> is the 0-based ascending sample subscripts (the Rust native convention, the R wrapper adds +1 before indexing the sample names); the cutoff crosses as an integer-valued double (the u64 domain, exact to 2^53 for an R double); cluster_stats = List{ retained_mean, retained_sd, retained_n, n_fits, converged } the retained-set diagnostics (the cutoff-plot hooks). manual_cutoff/seed are integer-valued doubles in [0, 2^53] (re-checked on the Rust side) – an explicit widening over the task’s minimal signature (counts, n_threads): the cutoff derivation and the seeding are inside the kernel, and if the seed/manual_cutoff do not cross the boundary the twin parity is moot (needed for freezing the semantics of the R API’s manual_cutoff/seed). threads/interrupt (contracts 6/7, a documented decision): a single cutoff rule = a fixed-order computation over n scalars (EM ≤200 iterations, pruning ≤n rounds, sub-millisecond) – nothing to parallelize (A7 trivially thread-invariant), no chunk structure to poll, declared bounded-duration uninterruptible (the same decision as ms_extract_rust); n_threads is only validated ≥0 without building a pool. The FFI internal name: the user API is ms_stratify_hypermutants() (R/stratify.R switched its main path to the FFI, the pure-R protocol twin is kept as the msuiter_stratify_twin() reference implementation + tests) 1, 2, 3, 4, 5 kernel 0.0.0.9000
ms_ard_rust (counts: R double matrix = t(counts), k0: i32, max_iter: i32, tol: f64, a0: f64, b0: f64, seed: i32, n_threads: i32) -> List .ms_ard_rust(counts, k0, …) the ARD-NMF kernel (U-M2-04, the SignatureAnalyzer semantics, the L1W.L2H variant + the automatic rank): returns the RAW signatures m×k_est column-major / exposures k_est×n / active / k_est / beta / beta_cut / objective / iterations / converged; k_est ≤ k0 (pruned away by β) 1, 2, 3, 4, 5 kernel 0.0.0.9000
ms_sparse_rust (counts = t(counts), k: i32, max_iter: i32, variant: String, lambda/mu/delta: f64, tol: f64, seed: i32, n_threads: i32) -> List .ms_sparse_rust(counts, k, …) the penalized sparse NMF kernel (U-M2-05): variant="volume" (the Leplat–Gillis log-det volume regularization, delta=0 takes the kernel default 1) / "l1" (the SparseSignatures L1L1, mu>0 required); returns the RAW signatures m×k / exposures k×n / objective / iterations / converged 1, 2, 3, 4, 5 kernel 0.0.0.9000
ms_fit_rust (counts: R double matrix = t(counts), n_samples×m_channels, signatures: R double matrix = t(sigs), k×m_channels, method: String, nb_size: f64, tol: f64, max_iter: i32, zero_threshold: f64, n_threads: i32, connected_components: Vec<i32> (may be empty = NULL/empty), rescale: bool) -> List .ms_fit_rust(counts, signatures, method, nb_size, tol, max_iter, zero_threshold, connected_components = NULL, rescale = TRUE, threads) (R/fit.R, internal) the reference fitting kernel (U-M3a-02, the L-D three methods): ① "nnls" = per-sample Lawson–Hanson (the K-format G = SᵀS, b_j = Sᵀv_j, the engine/nnls.rs semantics called with zero changes, the deconstructSigs baseline); ② "likelihood_bidirectional" = the MuSiCal nnls_likelihood_bidirectional semantics (research/02 §1.1: the all-catalog NNLS initial support -> the backward step Δ_j = L_cur − L_(−j) on the per-mutation multinomial LL scale (the LL_EPS=1e-16 clip, i.e. the upstream clip), removing the argmin while < ϵ (ϵ=tol, the frozen default 0.001) -> the forward step symmetrically adding -> alternating to both stops, the max_iter round cap (the frozen default 1000 = the upstream anti-cycling cap)); ③ "lrt" = the mSigAct flavor – the NB-MLE exposures (the NNLS warm start + the internal safeguarded-Newton projected ascent: the curvature block SᵀDS via the engine/linalg LDLᵀ, falling back to the steepest ascent when singular, the projected-gradient KKT + the relative-improvement dual stop, deterministic, zero external optimizer D9) + the per-signature entry-wise presence LRT (ll_full vs the target’s NB refit without it, stat = engine::likelihood::lrt_stat, p = ½erfc the Self–Liang default, the raw p without BH – the BH belongs to U-M3a-03’s ms_test_presence; the PASA forward set search is not in this unit). The unified evidence columns – the lrt_stat/lrt_p grid is method-independent and byte-consistent (always computed on the NB-MLE exposures from the full NNLS warm start, the presence evidence does not depend on the exposure estimator); support = the zeroing decision over the returned exposures (the zeroing_mask_share semantics: share < zero_threshold strictly -> exactly set to 0 + marked 0, the threshold parameterized through the FFI, the frozen default 0.01 = the sigfit-family anchor); se = the Fisher SE of the post-zeroing exposures (engine::likelihood::fisher_se, the interior support only, the boundary NaN = “boundary parameters get tests, not intervals”). The layout contract – the same as ms_extract_rust extended to the dictionary: R passes t(counts) (the n×m column-major buffer = the row-major m×n counts) and t(sigs) (the k×m column-major buffer = the row-major m×k signatures), zero marshalling on the Rust side; the transpose guard = the asymmetric golden discipline (both the Rust fit.rs and the R-end acceptance smoke assert the reconstruction cosine ≥ 0.99 over a distinguishable ground truth at m≠n). The validation (contract 4) – the counts cellwise finite (na) / non-negative + integer-valued (argument), the signatures cellwise finite/non-negative, nb_size finite>0, tol finite≥0, max_iter≥1, zero_threshold∈[0,1], m/n/k≥1, the channel dimensions of the two matrices agreeing; the error payload carries the 1-based natural coordinates (counts: channel × sample; signatures: channel × signature). threads/interrupt (contracts 6/7, a documented decision) – a single call = a per-sample bounded fixed-order batch (each inner loop capped: the bidirectional rounds / the NNLS active-set / the line search 80 halvings), declared bounded-duration uninterruptible (the same decision as ms_extract_rust); n_threads is only validated ≥0 without building a pool (A7 trivially thread-invariant; the sample parallelism belongs to U-M3a-03’s bootstrap). The U-M3a-05 closing two post-steps (the research/02 §1 row 27 the TMB recalibration + the row 23 the connected rejoin anchor) – ① rescale: bool (the TMB recalibration, fit::rescale_to_totals): per sample multiplying back the original mutation totals totals[j]/Σ_a h[a,j] (the conservation nailed at 1e-12; an all-zero column stays all-zero, a factor 0 clears the column exactly; only multiplying, never moving – a zeroed zero does not revive); ② connected_components (may be empty, the 0-based dictionary indices, converted by the wrapper from the 1-based R-side API): non-empty means ConnectedSpec{strategy: FixAndRefit} – the connected component (the clock-like background group) is fixed at the initial all-dictionary NNLS solution after the method solves, the remaining components re-solve the NNLS against the residual b_f − G_fc·h_fix (the group-alternating minimization, the frozen block converging in a single round). The D13 deviation declaration: upstream MuSiCal re-adds the connected signatures into the final NNLS support and jointly re-solves all the part loads (research/02 §1 row 23); FixAndRefit pins the connected block at the initial solution, not adjusting it with the sparse refit – a documented approximation, not the exact upstream semantics; ConnectedStrategy::JointRefit (the exact upstream joint re-solve) is declared future work, requesting it yields a structured argument error, never a silent substitution. The pipeline order: the method exposures -> the connected rejoin -> the TMB recalibration -> the share zeroing decision (the share is per-sample scale-invariant ⇒ the rescale does not flip a zeroing verdict; the rejoin does) -> the Fisher SE; the unified evidence columns lrt_stat/lrt_p are computed before the post-steps and are unaffected by them (byte-wise). With no post-step (NULL + false) the frozen fit::fit face runs byte-equivalent to the old pipeline (nailed). Returns List{ exposures k×n column-major (exact 0 after the zeroing), support k×n column-major i32 0/1, lrt_stat/lrt_p/se k×n column-major (the se boundary NaN), method (the echoed normalized string), converged (the method's internal convergence ∧ the per-sample AND over all the presence NB-MLE convergences) } – the label assembly, the tidy tables and the adhoc reference_snapshot belong to the R layer. The FFI internal name: the user API is the ms_fit() generic (R/fit.R, the MsCatalog method + the class_any fallback; the MsSignature sugar; the default "nnls" – the MSU-Fit default orchestration belongs to a later unit) 1, 2, 3, 4, 5 kernel 0.0.0.9000
ms_fit_bootstrap_rust (counts: R double matrix = t(counts), n_samples×m_channels, signatures: R double matrix = t(sigs), k×m_channels, method: String, n_boot: i32, nb_size: f64, zero_threshold: f64, seed: i32, n_threads: i32, bca: bool) -> List .ms_fit_bootstrap_rust(counts, signatures, method, n_boot, nb_size, zero_threshold, seed, threads, bca) (R/fit.R, internal) the nonparametric bootstrap CI kernel (U-M3a-03, the L-D uncertainty axis, the research/02 §2 semantics “resample the mutations (multinomial), not the channels”): per boot b, per sample j, engine/resample.rs::multinomial resamples N_j mutations from sample j’s empirical channel distribution (exactly N_j stream words, a zero-weight channel exactly zero counts), the boot stream = StreamId{replicate: b, rank: 0, fold: 0} (the canonical layout v1; the n sample resamplings within a boot are consumed sequentially on the same stream in the fixed sample order – resample.rs “composing several resamples on one stream is transparent”, a boot is a pure function of (seed, b, counts)); each resampled catalog is refit through fit::method_exposures (the per-sample exposure core shared by the three ms_fit methods, the U-M3a-02 semantics called with zero changes) -> the percentile CI (the type-7 linear interpolation, i.e. the R quantile default; the levels frozen at 2.5%/97.5% = BOOTSTRAP_CI_LEVELS) + support_stability (the frequency of support=1 across the boots). The bidirectional method’s ϵ and iteration caps are the frozen hyperparameters fit::FIT_EPS=0.001/fit::FIT_MAX_ITER=1000 (the same source as the R-side .ms_fit_defaults, the FFI surface carries no tol/max_iter – the Delta discipline, refreeze both sides together). The parallelism/interrupt (contracts 6/7) – the parallelism is only between the boots (the independent units): chunked over the per-call pool (≈4/worker, cap 16), the main thread pols R_CheckUserInterrupt at the chunk boundaries, the workers only see the AtomicBool; the chunks are concatenated by index and each element is computed by exactly one unit from its own stream – the outputs for threads∈{1,N} are identical() (nailed on the Rust side); the wire format (the compression decision, this document is the declaration) – returns the CI summary List{ ci_lower/ci_upper/support_stability k×n column-major, n_boot i32, method echoed, converged (the AND over all the boots converging) }, not returning the raw n_boot×k×n cube (bounded memory + the percentile arithmetic nailed by the tests). The validation is as in ms_fit_rust (contract 4: the matrix cellwise + the nb_size/zero_threshold domains + n_boot≥1/seed≥0/n_threads≥0; a sample total beyond the 2^53 exact-integer domain -> argument). The FFI internal name: the user API is the ms_fit_bootstrap() generic (R/fit.R, the MsCatalog method + the class_any fallback; MsFit carries the results through the CI properties + the support table’s boot_stability column). The U-M3b-02 BCa face (the bca parameter, the default false = percentile-compatible, true = BCa) – the same boot sample yields the BCa intervals with zero extra resampling (Hall 1988 doi:10.1214/aos/1176350930; the M3b design memo §1.2 the PI adjudication: BCa the default, the percentile the degeneration fallback): z₀ = Φ⁻¹(#{θ*_b < θ̂}/B) (the h₀ interior clamp [1e-8, 1−1e-8] – one-sided boots keep z₀ bounded at ±5.62 rather than ±∞; the clamp instead of the “+1e-8” sketch because the latter yields NaN at h₀=1); the acceleration = the fixed-active-set analytic jackknife (engine/bca.rs::analytic_acceleration: under the interior linear map h_A = G_A⁻¹S_Aᵀv, dropping one mutation gives exactly θ_(−mut@c) = θ̂ − u_c, u_c = G_A⁻¹ s_A[c], one pivoted LDLᵀ solve per sample over all the channels, the Efron-weighted central moment â = Σv_c(u_c−ū)³/(6(Σv_c(u_c−ū)²)^{3/2}), the weights = the channel counts; a singular G_A / an empty support / a zero-count sample -> that sample’s cell falls back) -> the corrected level α = Φ(z₀+(z₀+z_α)/(1−â(z₀+z_α))) (engine/bca.rs bca_levels; Φ via the likelihood::erfc path, Φ⁻¹ = Acklam + a Halley polish ~1e-13, nailed against the R qnorm anchor). The degeneration fallbacks (the thresholds frozen as the engine/bca.rs constants): z₀ <0.1 ∧ â
ms_test_presence_rust (counts: R double matrix = t(counts), n_samples×m_channels, signatures: R double matrix = t(sigs), k×m_channels, nb_size: f64, n_threads: i32) -> List .ms_test_presence_rust(counts, signatures, nb_size, threads) (R/fit.R, internal) the cohort-level per-signature presence test + the BH multiple-correction kernel (U-M3a-03, the research/02 §2 mSigAct “χ²₁ + BH q” semantics): ms_fit’s per-sample entry-wise presence LRT machinery (method-independent – the unified evidence columns) summed and pooled across samples D_a = Σ_j D_{a,j} (the independent likelihoods added; each signature accumulated in the sample-ascending fixed order), the raw p = the exact pooled null distribution’s strict upper tail p = Σ_{m=1}^{n} C(n,m)/2ⁿ·P(χ²_m > D) (the 2026-10 independent-audit P0 fix: under H0 each sample’s D_j ~ ½δ₀+½χ²₁ the Self–Liang boundary law, the independent catalogs added ⇒ the exact null distribution of the pooled D = the Binomial(n,½)-χ²_m mixture (the m=0 atom term P(δ₀>D)=[D<0], which is 0 for every admissible D≥0); the old reference – evaluating a single χ²₁ half-tail on the pooled D – is exact only for n=1 and anti-conservative for n≥2 (the measured pass_bh under H0 0.117/0.261/0.557/0.920 @ n=2/4/8/16); for n=1 it byte-degenerates to the ½·erfc(√(D/2)) half-tail (Self–Liang); the D13 caliber: this is a correction of the χ² reference and not a new statistical claim – the pooled estimand, the per-sample lrt_p column, the BH and the pass_bh logic are all unchanged; the weights’ implementation: for n≤53 the exact binary weights C(n,m)/2ⁿ (dyadic exactly representable), for n>53 the ln Γ recurrence + the weight floor e⁻⁷⁰⁰ skipping (the skipped mass <1e-12, nailed by the tests); the χ²_m upper tail = the regularized incomplete Gamma Q(m/2, D/2), df=1 via the engine erfc path, df≥2 the series/continued fraction, the quantile anchors nailed at 1e-10), the BH = a self-implemented step-up (sorting the p ascending, multiplying the i-th smallest by m/i, accumulating the minimum “monotone fold-back” from the largest p, capped at 1; the stable sort = the first index breaking a tie; adj ≥ raw and the fold-back monotonicity both nailed) → pass_bh (p_bh ≤ the frozen fit::BH_ALPHA=0.05; the raw/adjusted p also returned for re-thresholding). The NB-MLE refit uses the frozen fit::FIT_MAX_ITER (the ms_fit default). threads/interrupt (contracts 6/7, a documented decision) – a bounded fixed-order batch (the same ms_fit_rust decision: each inner loop capped), declared bounded-duration uninterruptible; n_threads is only validated ≥0 without building a pool (A7 trivially thread-invariant). Returns List{ lrt_stat/lrt_p_raw/lrt_p_bh k, pass_bh k (bool), converged (the AND over every sample's full/reduced NB-MLE convergence) }. The FFI internal name: the user API is the ms_test_presence() generic (R/fit.R: the MsCatalog face = this kernel; the MsFit face = the R-side same-semantics aggregation over the fit@tests grid – the stat summed, the exact pooled null distribution tail via the pure-R twin .ms_presence_pooled_p (the dbinom weights × the pchisq upper tail, the m=0 atom term handled explicitly – R’s pchisq(x, df=0, lower.tail=FALSE) uses a different convention at x=0; for n=1 byte-equal to the ½·pchisq half-tail), the pure-R BH twin .ms_bh_adjust agreeing with the kernel (the BH byte-wise; the p values are the same mathematical quantity on both sides, the cross-face agreement <1e-8, nailed by the tests)) 1, 2, 3, 4, 5 kernel 0.0.0.9000
ms_calibration_grid_rust (signatures: R double matrix = t(sigs), k×m_channels, n_grid: Vec<i32>, shares: Vec<f64>, n_samples: i32, n_reps: i32, n_boot: i32, nb_size: f64, bca: bool, arm: String, seed: i32, window_lo: f64, window_hi: f64, n_threads: i32) -> List the R-face assembly U-M3b-04: the exported function ms_calibration_grid() (the R/calibrate.R internal wrapper .ms_calibration_grid_rust() calls this internal face and assembles the verdict-window / confusion-tally curve table; plot_calibration_curve() renders the verdict curve, R/calibrate-plot.R) the calibration Monte Carlo driver (U-M3b-03, the M3b design memo §1–§3, the PI adjudication 2026-10-04): the n_grid × shares grid (the cell order = n_grid major × shares; a single call sweeps one true dictionary – the entropy axis is calling once per dictionary) × the frozen true-value convention (cell_spectrum: signature 1 carries the swept share, the remainder split evenly over the other signatures, the last element taking the exact residue so the spectrum sums to exactly N; share=0 is the decoy = the estimand-③ FPR point) × the dual generative arms arm ("multinomial" the main arm = exactly N stream words multinomially resampled / "nb:<κ>" the Jiang stress arm = per-channel independent NB / "poisson" = the κ=∞ exact limit; the two arms reported separately, never mixed into one calibration claim) × n_reps MC replicates per cell. The stream-layout extension (memo §2.1) – the MC stream = StreamId{replicate: rep, rank: the 1-based cell_id (the NB arm offset + n_cells), fold: 0}, disjoint from the bootstrap (rank=0) / CV (fold≥1); the first word of each replicate’s MC stream = the inner bootstrap’s master seed (a pure-function derivation, seed → the whole grid byte-wise). Per replicate: drawing the counts -> the production-face fit (fit::fit + fit::bootstrap: likelihood_bidirectional, the frozen FIT_EPS=0.001/FIT_MAX_ITER=1000, the share zeroing 0.01, the rescale aligned, the optional BCa face) -> the coverage verdict (estimand ②: the true h*_a ∈ the chosen CI face, the per-support-signature marginal tally) + the compound-zeroing confusion tally (estimand ③, the memo §3 frozen rules: the point support=0 ∨ stability<0.95 ∨ ci_lower_share<0.01; true value × decision → true_zero / false_keep (the FPR direction) / false_zero (the FNR direction) / kept_covered / kept_missed, the decoy entering the four cells but not the coverage). The parallelism/interrupt (contracts 6/7) – the parallelism is only between the (cell, replicate) units (a local copy of the fit.rs chunk skeleton: ≈4/worker chunks, the main-thread boundary poll R_CheckUserInterrupt, the workers only see the AtomicBool, the chunks concatenated by index); single-threaded within a unit (the inner bootstrap on 1 thread) → the outputs for threads∈{1,N} are identical (nailed on the Rust side). The validation (contract 4): n_grid/shares non-empty, the n_grid entries ≥1, share∈[0,1] finite, n_samples/n_reps/n_boot ≥1, seed/n_threads ≥0, the t(sigs) cellwise finite/non-negative, the spectrum sum positive and near-integer, nb_size finite>0, the arm name legal (illegal -> argument listing the legal names). Returns List{ mean_coverage/se (per cell, se=√(p(1−p)/tallies)), n_reps, the n_grid/shares/arm/bca/n_boot/nb_size/seed echoes, true_zero/false_keep/false_zero/kept_covered/kept_missed (the per-cell f64 tallies), window_lo/window_hi/all_in_window/per_cell_in/failing_cells (the 0-based cell indices) } – the verdict window inclusive of the endpoints, the window bounds passed by the caller (the PI default [0.930, 0.970] = calibrate::CALIBRATION_WINDOW, the 4SE construction at M=2000 giving 4·√(0.95·0.05/2000)≈0.0195). The test anchors: a perfect-model cell’s coverage≈1; the M=500 reduced protocol’s percentile arm within the [0.930,0.970] window; the BCa arm as a control; the grid’s seed determinism + the threads∈{1,3} identical; the share=0.005 boundary cell false-zero-dominated, the share=0 decoy true-zero-dominated; the NB arm’s κ-monotone undercoverage (the Poisson limit undercovering because the per-channel sampling does not conserve the total – the arm B not being conservatively covered is the memo §2.3 expected deliverable) 1, 2, 3, 4, 5, 6, 7 diagnostic 0.0.0.9000
ms_match_solutions_rust (estimated/reference: f64 vectors, dim: i32, thresholds: f64 vectors) -> List .ms_match_solutions_rust(estimated, reference, dim, thresholds) the M4 data face (memo §1.5): one cosine matrix × the threshold sweep – per threshold a Hungarian assignment + the TP/FP/FN counts + the per-estimated-signature classification (matched/split/merge/novel + the 1-based reference index); the Hungarian assignment semantics are shared by the M2 Islam reproduction and the M4 ms_compare (M4’s Islam greedy is an independent R implementation – the zoo §1.20 literal caliber, a different caliber from the M2 face’s split/merge refined tally, and the head-to-head bench proper is pending a PI adjudication) 4, 5 kernel 0.0.0.9000
ms_compare_null_rust (family: String, m: i32, burden: f64, n_draws: i32, quantiles: f64 vectors, seed: i32) -> List .ms_compare_null(family, m, burden, n_draws, quantiles, seed) (R/ms-compare.R, internal; the frozen MC-SE tolerance 0.01 enforced on the R face, exceeded -> msuiter_error_calibration) the calibrated-similarity-library null-distribution face (U-M4-01, the design memo vignettes/articles/devlog-2026-10-04-M4-01-design-memo.qmd §2/§5): two frozen null families – "signature_uniform" = the cosine between two independent uniform Dirichlet(1…1) signatures (= normalized i.i.d. Exp(1), the engine’s new primitive resample::dirichlet_uniform) (the “uncorrelated signatures by the channel count” null; the analytic anchors E⟨x,y⟩=1/m, E‖x‖²=2/(m+1), the delta approximation E[cos]≈(m+1)/(2m) – nailed at the test layer; the 0.75 chance baseline of Islam 2022 is a different family’s construction and explicitly not an anchor); "catalog_multinomial" / "catalog_poisson" / "catalog_nb:<κ>" (the arm syntax shared with the calibration driver’s GenerativeArm) = the reconstruction cosine at the burden N of the true μ (drawn once per call from the same uniform family and fixed) (the “by channels × burden” null calibration; monotonically rising as N→∞, the NB arm shifted down under stress – nailed). The stream layout – StreamId{replicate: 0, rank: the family band 1/2/3, fold: 0}, disjoint from the bootstrap (rank=0)/CV (fold≥1)/MC calibration (replicate≥1); μ drawn first then each draw, the whole result a byte-function of the seed. Returns List{ mean, sd, mc_se (the batch means, 20 batches, the i.i.d. batch-mean se=sd/√B), quantiles (in the caller's order, the type-7 empirical quantiles, the same convention as R's quantile(type=7) -- the audit measured a ≤2 ulp difference (R's FMA contraction), not claiming byte-for-byte), family/m/burden/n_draws/seed echoes }. threads/interrupt (contracts 6/7, a documented decision) – a bounded sequential loop (n_draws has the lower bound 200), declared uninterruptible and building no pool (same as ms_test_presence_rust). The validation (contract 4): m≥2 (the cosine degenerates at m=1), n_draws≥200, the catalog families’ burden≥1 finite, quantiles∈(0,1) finite non-empty, seed≥0; an all-zero catalog / a zero norm for the NB arm → the structured "na" error (the 1-based draw index), the quantiles never NaN. The FFI internal name: the user API is ms_compare() (R/ms-compare.R assembling the four metrics + the Hungarian/Islam/Jiang three protocols + the null distributions) 1, 2, 3, 4, 5 kernel 0.0.0.9000
ms_bootstrap_draws_rust (counts: R double matrix = t(counts), n_samples×m_channels, signatures: R double matrix = t(sigs), k×m_channels, method: String, n_boot: i32, nb_size: f64, zero_threshold: f64, seed: i32, n_threads: i32, bca: bool) -> List .ms_bootstrap_draws_rust(...) (R/fit.R, internal; ms_fit_bootstrap(return_draws = TRUE) routes here) the U-M7-pre draws face: the very same bootstrap run as ms_fit_bootstrap_rust (byte-equal with the same seed), additionally returning the RAW boot-exposure cube – draws = the n_boot×(k·n) matrix, the boot blocks row-major (draws[b·k·n + a·n + j] = the (signature a, sample j) exposure of boot b, post-zeroing/post-rescale – the very batch the CI/BCa reduction reads). The remaining members (ci_lower/ci_upper/support_stability/n_boot/method/converged) are byte-consistent with the interval face. The guard: k·n·n_boot > 2e7 doubles -> the structured argument error (the checked arithmetic guarding overflow). The layout trap (the audit note): the R-side reshape must t() first then array(dim=c(n,k,n_boot)) + aperm – the double trap of the R matrix’s column-major flattening and the array fill order, guarded by the type-7 quantile parity nail (≤2 ulp, R’s FMA). The interrupt/threads (contracts 6/7) as on the interval face 1, 2, 3, 4, 5, 6, 7 kernel 0.0.0.9000
ms_compare_shape_null_rust (profile: R double vector (length m, shares or counts -- auto-normalized), burden: f64, n_draws: i32, quantiles: f64 vectors, seed: i32) -> List .ms_compare_shape_null(profile, burden, n_draws, quantiles, seed) (R/ms-compare.R, internal; the same frozen 0.01 MC-SE tolerance gate) the U-M7-pre shape-conditional null (the concretization of the memo’s “entropy-matched Dirichlet(α·u)” upgrade arm): draws = Multinomial(N, u)/N, the statistic = cos(draw, u) – “can this deviation be explained by the sampling noise alone, at this shape and this burden”. The entropy semantics enter naturally: the shape through u, the noise scale through α_total=N (the multinomial’s continuous limit, no α dichotomy needed); a flat reference automatically yields a tighter null (the Medo flat narrative). The stream layout StreamId{replicate: 0, rank: 4, fold: 0} (the reserved slot of the null band). A different law from the catalog face (the audit-style finding: the catalog’s marginalized μ concentrates from Dirichlet(1..) → a higher cosine; the uniform conditioning is the flattest; the direction assertion + the χ²/N scaling-law anchors – the sd ratio ≈ 1/N the ratio 10, not Gaussian’s 1/√N). The validation: m≥2, profile finite positive and not all-zero (the na tier), n_draws≥200, burden≥1 finite, quantiles∈(0,1). threads/interrupt: bounded sequential uninterruptible (as the other null faces). The test anchors: the burden monotonicity, the direction + the scaling law, the determinism, the profile counts auto-normalized 1, 2, 3, 4, 5 kernel 0.0.0.9000
ms_pipeline_rust | (counts: R double matrix = t(counts), n_samples×m_channels, k: i32, replicates: i32, max_iter: i32, seed: i32, k_folds: i32, n_seeds: i32, n_threads: i32) -> List | .ms_pipeline_rust(counts, k, replicates, max_iter, seed, k_folds, n_seeds, threads) (R/extract.R, internal) | the default consensus-CV extraction pipeline kernel (U-M2-03, the FFI face of the D16 default path): one entry one exit, five phases – ① the ensemble (the U-M1s-05 driver extended nmf_ensemble: the replicates independent KL-NMF fits in parallel over the per-call pool, replicate r’s seeded init from StreamId{replicate: r, rank: 0, fold: 0}, returning per replicate the factor pair (W,H) – the same stream, the same scheduling and the same interrupt protocol as the objective path, only the return differs; the ensemble axis = multiple initializations only, no bootstrap, the PI adjudication 2026-09-30); ② the stacking (each replicate’s W columns normalized sum-1 then stacked = the upstream pnmf semantics; a zero column stays zero, the zero-vector cosine convention absorbed); ③ the Hungarian consensus (the engine::consensus::consensus_cluster semantics called with zero changes: the 50-restart iteration re-assignment + the silhouette convergence, StreamId{replicate: restart, rank: k, fold: 0}, bounded sequentially within the main thread); ④ the per-sample NNLS refit (the engine::nnls::nnls_gram K-format: G = WᵀW, b_j = Wᵀv_j, solving the non-negative exposure per sample against the consensus centroid W, the upstream fit_signatures_pool step); ⑤ the SUITOR CV evidence (the engine::cv::cv_error semantics called with zero changes: the rank grid 1..=k, StreamId{replicate: seed, rank, fold≥1} – disjoint from the ensemble (fold 0)/consensus-restart (fold 0) streams; the seeds default 10 = the PI adjudication divergence, upstream 30). The layout contract – same as ms_extract_rust: R passes t(counts), the n×m column-major buffer is the row-major m×n V, zero marshalling on the Rust side. The validation (contract 4): k∈[1, min(m,n)] (i=k, j=min(m,n)), replicates≥2 (the consensus silhouette needs two members per cluster), max_iter≥1, k_folds∈[2, m], n_seeds≥1, seed/n_threads non-negative; the content validation (finite/non-negative) is structurally thrown by the kernel’s first unit (the 1-based i/j). threads/interrupt (contracts 6/7): ① follows the replicates pattern polling R_CheckUserInterrupt at the chunk boundaries + the workers only see the AtomicBool; the ③④⑤ phases are bounded sequential computations (declared: the ensemble phase dominates the wall clock, the later phases uninterruptible – the same documented decision as ms_extract_rust). **Returns List{ consensus_W m×k column-major (the RAW consensus centroid), stability_per_cluster k, avg_stability, nnls_exposures k×n column-major (the RAW refit, against the RAW W), cv_per_rank k (the CV.te for rank 1..=k, NaN = no finite fold for that rank, mapped to NA on the R side), cv_per_rank_train k, fold_test_deviance k×k_folds (NaN = the fold has no finite seed), argmin_rank i32 (0 = no finite rank total), consensus_best_restart i32, consensus_n_rounds i32, consensus_converged bool } – the W column normalization and the label assembly belong to the R layer (the refit exposure against the RAW W scales by the same factor as W, the reconstruction conserved). The FFI internal name: the user API is ms_extract(method = NULL) (the D16 default-path assembly) and ms_select_k() (the K-adjudication grid evidence, R/kselect-select.R) | 1, 2, 3, 4, 5, 6, 7 | kernel | 0.0.0.9000 |

Conventions:

  • The status has only two tiers: diagnostic (this unit’s contract probes / meta-information, no domain algorithm) and kernel (a real algorithm-kernel export, entering from M1s). Inside the M0 milestone no kernel export exists – this is deliberate (U-M0-09 forbids implementing any real algorithm kernel). The U-M3b-03 exception (the 2026-10-04 task adjudication; the semantics revised from U-M3b-04): although ms_calibration_grid_rust contains domain algorithm, its essence is the orchestration-diagnostic face of the M3b calibration bench (the numerical substance is carried by the ms_fit/bootstrap kernel faces; from U-M3b-04 the user-side exported function ms_calibration_grid() calls this face through an internal wrapper – the R face is the assembly layer, this face is not exposed directly to calls outside the driver), so it is recorded in the diagnostic tier – from then on the tier’s semantics subdivide to “contract probes / meta-information + the bench diagnostic faces”.
  • The since = the package version at which the export first appeared; once published it is never rewritten. (See the version-stamp-semantics note in the header: the frozen M0-generation artifact seals every row at the M0 stamp 0.0.0.9000; per-export introduction history lives in the rows’ unit-ID references and the devlogs.)
  • The error protocol: every msffi_*’s Err(MsError) is uniformly converted at the ffi boundary to a condition object of class c("msuiter_error_rust", "error", "condition") (the payload message, topic, i, j, c), re-raised by the R wrapper via stop(); the abort classes of the R-side validators are msuiter_error_input / msuiter_error_na / msuiter_error_option (the msuiter_error_<topic> convention of ARCHITECTURE §3.5). The helper unification completed (2026-09-28, the U-M1s wrap-up small cleanup unit): every R-side abort (the six spots of .ms_tally_rust, .ms_validate_matrix/_count/_switch, .ms_resolve_threads and the three spots of .ms_extract_rust) is now uniformly emitted through the shared msuiter_abort() (R/classes-utils.R), the earlier temporary rlang::abort(context=) shape no longer exists. The literal context field and anyNA’s integer i/j coordinates are kept as top-level condition fields per the ffi contract – injected through the msuiter_abort(data=) compatibility parameter (rlang reserves c for the sub-conditions, hence not using c= directly), the canonical i/j/c three-segment bullets carried in cnd$msuiter_error; the err$i/err$j/err$context assertions nailed by the tests keep passing unchanged.
  • The statelessness (contract 1) applies to every export: no handle registry table, no .state, no silent recomputation; the per-call thread pool is built and destroyed with the call.

The frozen protocol

  1. Any change to this manifest (adding an export, removing one, changing a signature, changing the error protocol) must:
    • update §“the export manifest” and §“the export detail table” in lockstep (incl. since and the status);
    • keep the drift-guard test green under devtools::test() (the manifest ↔︎ the generated wrappers set agreeing);
    • tick the FFI-surface check item in the PR description – the check item lands via U-M0-08’s PR template (see the ROADMAP’s M0 governance-files entry), before the template lands declaring “ffi-surface change” explicitly in the PR description substitutes for it;
    • when a ms_*_rust kernel export (M1s+) is involved, refresh the corresponding golden/fixture in lockstep (the PR template’s second check item, carried the same way).
  2. The msffi_* belong to the diagnostic layer: allowed to grow and shrink with review; the ms_*_rust belong to the kernel layer: once published they are frozen (a semantic change = a new export name + the old name deprecated through lifecycle::deprecate_warn).
  3. This file is reviewed once per release (the per-release review artifact, ARCHITECTURE §3.5); the review record goes into that period’s devlog.