carverauto/serviceradar

Fork 0

Unify the anomaly reasoner on DeepCausality (fold O(1) Welford into the NIF + reason_batch, delete the hand-rolled compact evaluator) #3796

New issue

Open

opened 2026-06-13 22:43:00 +00:00 by mfreeman451 · 2 comments

mfreeman451 commented

2026-06-13 22:43:00 +00:00

Owner

Summary

Unify ServiceRadar's per-series anomaly reasoner on DeepCausality (Marvin Hansen's library) as the single source of truth: fold an O(1) Welford rolling-statistics accumulator into the existing causal_reasoner_nif, add a batched reason_batch entry to amortize FFI, and delete the hand-rolled compact_evaluator.ex. Benchmarked and ready to implement.

OpenSpec proposal: openspec/changes/refactor-anomaly-reasoner-deepcausality (PR pending). AGENTS.md guideline added: use DeepCausality for causal/statistical/streaming-anomaly reasoning; do not hand-roll a parallel detector.

Problem

Two divergent implementations of the same rolling z-score exist:

causal_reasoner_nif — built correctly on DeepCausality (deep_causality_core::CausalFlow + deep_causality_data_structures::SlidingWindow), but recomputes mean/variance over the full window every sample (O(window)) and marshals the whole baseline list across FFI each call.
compact_evaluator.ex — a hand-rolled pure-Elixir Welford z-score, no DeepCausality, no production callers, added to chase throughput.

Two implementations of the same statistics in two languages must be kept in numeric parity by hand and drift. That already produced the PR #3792 variance bug (naive Σx² − (Σx)²/n collapsing stddev→0 on large counters), and the NIF's two-pass variance has the same f64 cancellation class at counter magnitudes.

Benchmarks (this hardware, OTP 28, 10 cores)

Elixir end-to-end:

path	evals/sec	what
`owner` (production: GenServer + NIF, O(window))	~50k	real today
`reasoner` (NIF only, O(window), per-sample FFI)	~140k	FFI-bound
`compact` (Elixir O(1), single core)	5.7–9.4M	bench-only
`compact_shards` (Elixir O(1), 10-core tight loop)	13.2M	best-case microbench
`compact_ets_shards` (Elixir O(1) + per-series ETS state)	2.0M	realistic distributed state

Rust microbench (/tmp/dc_bench, in-process, no FFI, W=300, 5M samples at 1e9 magnitude, the crates.io DeepCausality versions the NIF ships):

variant	ns/op	evals/sec/core	meaning
A	~354	~2.8M	DeepCausality `SlidingWindow` + O(window) two-pass recompute (today's NIF compute)
B	~9.8	~102M	DeepCausality `SlidingWindow` + O(1) Welford (proposed)
C	~9.6	~105M	raw ring buffer + O(1) Welford (floor, no DeepCausality)
D	~24.8	~40M	`CausalFlow` wrapper + O(1) Welford (idiomatic per-sample flow)

Findings:

DeepCausality is not the bottleneck — its SlidingWindow costs ~0.2 ns over a raw buffer (B vs C = 1.02×).
The cost is the O(window) recompute — A vs B = ~36×. Welford erases it.
FFI is the only real constraint — per-sample NIF caps ~1M/s (FFI ~1 µs/call); batching ~1k samples/crossing amortizes to ~1 ns/sample → compute-bound at ~100M/s/core. That's ~8× a whole 10-core Elixir shard, ~50× the realistic ETS-distributed path, ~2000× production.
Accuracy bonus — at 1e9, the naive two-pass variance diverges ~1e-6 from Welford via catastrophic cancellation; B/C/D agree to the bit. Welford fixes speed and accuracy.

Decision

Fold O(1) Welford into the DeepCausality NIF and amortize FFI with a batch entry — do not fork a second Elixir engine. One reasoner, on DeepCausality, faster than the two-implementation approach, eliminating the parity-maintenance burden.

Implementation plan (all recommended, to be implemented)

Welford in the NIF. Add WelfordAcc{count, mean, m2} + welford_add/welford_remove (West deletion, clamp m2 ≥ 0) ported byte-for-byte from compact_evaluator.ex. Keep the two-pass path behind a flag as the parity oracle.
O(1) admit. On window.filled() read SlidingWindow.first() (oldest = next evict) → welford_remove, then push, then welford_add; before fill, add only.
Incremental, stateless-across-calls contract. ReasonContext carries acc + bounded window_tail (not the full baseline list); ReasonVerdict returns next_acc + next_window_tail, folded into the value channel in finalize_detector_verdict (since finish() drops flow State). Rewrite evaluate_detector rolling signal to read (mean, stddev) from acc (variance = m2/(count−1), zero-variance guard) instead of vec() + sample_stats; admit in the branch_with clean arm (withholding).
reason_batch(Vec<(SeriesState, Sample)>) -> Vec<Verdict> (regular NIF if per-batch < ~1 ms, else DirtyCpu). Batch on the Broadway/shard boundary, not per series; tune ~1k. Keep reason/2 for single-shot and rebuild_from replay.
Elixir state shrink, sharding unchanged. ContextOwner.reason_update calls the NIF with the accumulator context; persist {count, mean, m2, window_tail, consecutive} instead of a baseline list. Keep the ordered sample log only for out-of-order rebuild_from (replay through reason_batch). Horde/single-writer/Broadway concurrency untouched.
Keep CausalFlow per-sample (idiomatic, 40M/s, ~20× the target, home for future seasonal/trend/corrective causaloids); drop to the bare loop only if profiling later shows the ~15 ns matters.
Parity + drift gates. Keep the existing Rust test oracle green; add an incremental-vs-two-pass property test over random in-order streams incl. ≥1e9 magnitude; add a periodic full-slice recompute (or Kahan) to bound windowed-remove drift (simulated |Δz| ≈ 3e-5 over 200k evictions — negligible but bound it).
Remove the second implementation. Once at parity + acceptable batched throughput, delete compact_evaluator.ex, its test, and its bench modes; add a reason_batch bench mode and record results vs the documented numbers.

Notes / caveats

finish() returns only the value channel and drops State — the updated accumulator MUST be folded into the verdict value (mirrors how next_consecutive_anomalous is threaded out today). This is the one design point that bites if missed.
DeepCausality ships no stats primitive (verified: algorithms = BRCD/SURD/mRMR, metric = Clifford signatures, uncertain = Monte-Carlo batch) — we own the Welford math inside the causaloid; the win is the Flow + SlidingWindow primitives.
Relationship: refines update-anomaly-evaluation-cadence (its "compact incremental state" requirement is satisfied inside the NIF, not a separate Elixir evaluator) and consumes the counter rate/reset handling from add-monotonic-counter-metric-semantics unchanged.
Bench source preserved at /tmp/dc_bench/src/main.rs for re-runs.

Acceptance

Verdict parity (sample variance n−1, z-score + zero-variance guard, breach ≥ n-sigma, withholding, confirm-slots, insufficient-baseline gate, non-finite drop) verified by the Rust oracle + property test; reason_batch throughput recorded vs the compact and production paths; compact_evaluator.ex removed.

## Summary Unify ServiceRadar's per-series anomaly reasoner on **DeepCausality** (Marvin Hansen's library) as the single source of truth: fold an O(1) Welford rolling-statistics accumulator into the existing `causal_reasoner_nif`, add a batched `reason_batch` entry to amortize FFI, and **delete the hand-rolled `compact_evaluator.ex`**. Benchmarked and ready to implement. OpenSpec proposal: `openspec/changes/refactor-anomaly-reasoner-deepcausality` (PR pending). AGENTS.md guideline added: use DeepCausality for causal/statistical/streaming-anomaly reasoning; do not hand-roll a parallel detector. ## Problem Two divergent implementations of the same rolling z-score exist: - **`causal_reasoner_nif`** — built correctly on DeepCausality (`deep_causality_core::CausalFlow` + `deep_causality_data_structures::SlidingWindow`), but **recomputes mean/variance over the full window every sample** (O(window)) and marshals the whole baseline list across FFI each call. - **`compact_evaluator.ex`** — a hand-rolled pure-Elixir Welford z-score, no DeepCausality, **no production callers**, added to chase throughput. Two implementations of the same statistics in two languages must be kept in numeric parity by hand and drift. That already produced the PR #3792 variance bug (naive `Σx² − (Σx)²/n` collapsing `stddev→0` on large counters), and the NIF's two-pass variance has the same f64 cancellation class at counter magnitudes. ## Benchmarks (this hardware, OTP 28, 10 cores) **Elixir end-to-end:** | path | evals/sec | what | |---|---|---| | `owner` (production: GenServer + NIF, O(window)) | ~50k | real today | | `reasoner` (NIF only, O(window), per-sample FFI) | ~140k | FFI-bound | | `compact` (Elixir O(1), single core) | 5.7–9.4M | bench-only | | `compact_shards` (Elixir O(1), 10-core tight loop) | 13.2M | best-case microbench | | `compact_ets_shards` (Elixir O(1) + per-series ETS state) | 2.0M | realistic distributed state | **Rust microbench** (`/tmp/dc_bench`, in-process, no FFI, W=300, 5M samples at 1e9 magnitude, the crates.io DeepCausality versions the NIF ships): | variant | ns/op | evals/sec/core | meaning | |---|---|---|---| | A | ~354 | **~2.8M** | DeepCausality `SlidingWindow` + O(window) two-pass recompute (today's NIF compute) | | B | ~9.8 | **~102M** | DeepCausality `SlidingWindow` + O(1) Welford (proposed) | | C | ~9.6 | ~105M | raw ring buffer + O(1) Welford (floor, no DeepCausality) | | D | ~24.8 | ~40M | `CausalFlow` wrapper + O(1) Welford (idiomatic per-sample flow) | **Findings:** - **DeepCausality is not the bottleneck** — its `SlidingWindow` costs ~0.2 ns over a raw buffer (B vs C = 1.02×). - **The cost is the O(window) recompute** — A vs B = ~36×. Welford erases it. - **FFI is the only real constraint** — per-sample NIF caps ~1M/s (FFI ~1 µs/call); batching ~1k samples/crossing amortizes to ~1 ns/sample → compute-bound at ~100M/s/core. That's ~8× a whole 10-core Elixir shard, ~50× the realistic ETS-distributed path, ~2000× production. - **Accuracy bonus** — at 1e9, the naive two-pass variance diverges ~1e-6 from Welford via catastrophic cancellation; B/C/D agree to the bit. Welford fixes speed *and* accuracy. ## Decision Fold O(1) Welford into the DeepCausality NIF and amortize FFI with a batch entry — do **not** fork a second Elixir engine. One reasoner, on DeepCausality, faster than the two-implementation approach, eliminating the parity-maintenance burden. ## Implementation plan (all recommended, to be implemented) 1. **Welford in the NIF.** Add `WelfordAcc{count, mean, m2}` + `welford_add`/`welford_remove` (West deletion, clamp `m2 ≥ 0`) ported byte-for-byte from `compact_evaluator.ex`. Keep the two-pass path behind a flag as the parity oracle. 2. **O(1) admit.** On `window.filled()` read `SlidingWindow.first()` (oldest = next evict) → `welford_remove`, then `push`, then `welford_add`; before fill, add only. 3. **Incremental, stateless-across-calls contract.** `ReasonContext` carries `acc` + bounded `window_tail` (not the full baseline list); `ReasonVerdict` returns `next_acc` + `next_window_tail`, folded into the value channel in `finalize_detector_verdict` (since `finish()` drops flow State). Rewrite `evaluate_detector` rolling signal to read `(mean, stddev)` from `acc` (`variance = m2/(count−1)`, zero-variance guard) instead of `vec()` + `sample_stats`; admit in the `branch_with` clean arm (withholding). 4. **`reason_batch(Vec<(SeriesState, Sample)>) -> Vec<Verdict>`** (regular NIF if per-batch < ~1 ms, else DirtyCpu). Batch on the Broadway/shard boundary, not per series; tune ~1k. Keep `reason/2` for single-shot and `rebuild_from` replay. 5. **Elixir state shrink, sharding unchanged.** `ContextOwner.reason_update` calls the NIF with the accumulator context; persist `{count, mean, m2, window_tail, consecutive}` instead of a baseline list. Keep the ordered sample log only for out-of-order `rebuild_from` (replay through `reason_batch`). Horde/single-writer/Broadway concurrency untouched. 6. **Keep `CausalFlow` per-sample** (idiomatic, 40M/s, ~20× the target, home for future seasonal/trend/corrective causaloids); drop to the bare loop only if profiling later shows the ~15 ns matters. 7. **Parity + drift gates.** Keep the existing Rust test oracle green; add an incremental-vs-two-pass property test over random in-order streams incl. ≥1e9 magnitude; add a periodic full-slice recompute (or Kahan) to bound windowed-remove drift (simulated `|Δz| ≈ 3e-5` over 200k evictions — negligible but bound it). 8. **Remove the second implementation.** Once at parity + acceptable batched throughput, delete `compact_evaluator.ex`, its test, and its bench modes; add a `reason_batch` bench mode and record results vs the documented numbers. ## Notes / caveats - `finish()` returns only the value channel and **drops State** — the updated accumulator MUST be folded into the verdict value (mirrors how `next_consecutive_anomalous` is threaded out today). This is the one design point that bites if missed. - DeepCausality ships **no** stats primitive (verified: algorithms = BRCD/SURD/mRMR, metric = Clifford signatures, uncertain = Monte-Carlo batch) — we own the Welford math inside the causaloid; the win is the Flow + `SlidingWindow` primitives. - Relationship: refines `update-anomaly-evaluation-cadence` (its "compact incremental state" requirement is satisfied **inside the NIF**, not a separate Elixir evaluator) and consumes the counter rate/reset handling from `add-monotonic-counter-metric-semantics` unchanged. - Bench source preserved at `/tmp/dc_bench/src/main.rs` for re-runs. ## Acceptance Verdict parity (sample variance n−1, z-score + zero-variance guard, breach ≥ n-sigma, withholding, confirm-slots, insufficient-baseline gate, non-finite drop) verified by the Rust oracle + property test; `reason_batch` throughput recorded vs the compact and production paths; `compact_evaluator.ex` removed.

mfreeman451 referenced this issue

2026-06-13 22:43:24 +00:00

spec: unify anomaly reasoner on DeepCausality (single source of truth) #3797

mfreeman451 referenced this issue

2026-06-13 22:47:44 +00:00

spec: unify anomaly reasoner on DeepCausality #3798

mfreeman451 referenced this issue

2026-06-13 22:54:55 +00:00

spec: unify anomaly reasoner on DeepCausality #3798

mfreeman451 referenced this issue

2026-06-13 22:56:17 +00:00

spec: unify anomaly reasoner on DeepCausality (single source of truth) #3797

mfreeman451 referenced this issue

2026-06-13 23:01:35 +00:00

spec: unify anomaly reasoner on DeepCausality #3798

mfreeman451 commented

2026-06-13 23:04:48 +00:00

Author

Owner

Merged the OpenSpec proposal and AGENTS.md guardrail in #3798. Keeping this issue open because the NIF Welford state, reason_batch, runtime integration, parity/benchmark gates, and CompactEvaluator removal still need implementation.

Merged the OpenSpec proposal and AGENTS.md guardrail in #3798. Keeping this issue open because the NIF Welford state, `reason_batch`, runtime integration, parity/benchmark gates, and `CompactEvaluator` removal still need implementation.

mfreeman451 commented

2026-06-13 23:24:39 +00:00

Author

Owner

Implementation PR opened: #3799 (work/3796-deepcausality-impl -> staging).

This first implementation folds compact Welford rolling state into the DeepCausality NIF, adds reason_batch, persists compact state through ContextOwner, and marks the completed OpenSpec tasks.

Validation run:

cargo test in elixir/serviceradar_core/native/causal_reasoner_nif
MIX_ENV=test mix test test/serviceradar/observability/causal_reasoner_test.exs test/serviceradar/observability/anomaly_detection/context_owner_test.exs
MIX_ENV=test mix test test/serviceradar/observability/anomaly_detection/synthetic_dataset_test.exs
MIX_ENV=test mix format --check-formatted ...
cargo fmt --check
openspec validate refactor-anomaly-reasoner-deepcausality --strict

Remaining work is still tracked in the proposal tasks: batch/shard wiring, benchmark modes/results, and eventual CompactEvaluator cleanup after acceptance.

Implementation PR opened: #3799 (`work/3796-deepcausality-impl` -> `staging`). This first implementation folds compact Welford rolling state into the DeepCausality NIF, adds `reason_batch`, persists compact state through `ContextOwner`, and marks the completed OpenSpec tasks. Validation run: - `cargo test` in `elixir/serviceradar_core/native/causal_reasoner_nif` - `MIX_ENV=test mix test test/serviceradar/observability/causal_reasoner_test.exs test/serviceradar/observability/anomaly_detection/context_owner_test.exs` - `MIX_ENV=test mix test test/serviceradar/observability/anomaly_detection/synthetic_dataset_test.exs` - `MIX_ENV=test mix format --check-formatted ...` - `cargo fmt --check` - `openspec validate refactor-anomaly-reasoner-deepcausality --strict` Remaining work is still tracked in the proposal tasks: batch/shard wiring, benchmark modes/results, and eventual `CompactEvaluator` cleanup after acceptance.

mfreeman451 referenced this issue

2026-06-14 05:20:52 +00:00

perf: instrument anomaly production benchmark #3817

mfreeman451 referenced this issue

2026-06-14 05:20:52 +00:00

perf: combine anomaly event prep and shard grouping #3821

mfreeman451 referenced this issue from a commit

2026-06-14 06:39:42 +00:00

fix: address anomaly engine review findings (#3796 stack)

mfreeman451 referenced this issue

2026-06-14 06:41:06 +00:00

fix: address anomaly engine review findings #3828