carverauto/serviceradar

Fork 0

[anomaly-detection + capacity-forecast UI: state and surfacing (OpenSpec add-causal-anomaly-detection, Phases 4.4 + 5.1-5.4)][blocker] Detector and forecaster produce ZERO data in demo — spine not running, UI renders empty #3731

New issue

Open

opened 2026-06-13 05:14:23 +00:00 by mfreeman451 · 4 comments

mfreeman451 commented

2026-06-13 05:14:23 +00:00

Owner

Problem

The causal anomaly-detection and capacity-forecast UI surfaces (tile, dashboard visual, event findings) render empty in the live demo because the data-production spine that should feed them is not running at all in the demo deployment. Every UI element that was built (Phases 4.4 + 5.1-5.4 of OpenSpec add-causal-anomaly-detection) has nothing to show: the tile reads 0/0/0 and the dashboard panel shows "No capacity forecast data".

This is upstream of, and distinct from, the verdict-routing/alert-spine gaps in #3706 / #3710 / #3711 (which assume verdicts are being produced and then dropped in transit). Here, nothing is produced in the first place.

Severity

blocker — the entire feature produces zero data in the demo; the UI is empty and cannot be evaluated.

Root cause

The data-production spine is not running in the demo deployment:

Real-time anomaly detector defaults OFF. ANOMALY_ANALYSIS_CONSUMER_ENABLED defaults to "false" in runtime.exs, and it is not set on the demo core pod. The consumer (and AnomalyDetection.Supervisor) only starts when AnomalyDetection.Config.enabled?() is true, so the detector never runs.
Capacity-forecasting Oban cron has never enqueued a single job in demo. There are no CapacityForecasting.Worker rows in oban_jobs at all. The demo core pod has the config-seed env vars (SERVICERADAR_CAPACITY_FORECAST_CONFIG_*) but not SERVICERADAR_CAPACITY_FORECASTING_ENABLED. Even though the code default is "true", the absence of ANY worker job suggests the crontab plugin entry is gated out or the enqueue is not happening — the crontab wiring needs verification against the loaded Oban plugin config.
seasonal_profiles table does not even exist (relation "seasonal_profiles" does not exist), so the seasonal baseline the forecaster depends on cannot be built.

Source data IS present (cpu_metrics=231,112 / memory_metrics=115,556 in the last 3 days), so this is purely a "spine not running" problem, not a missing-input problem.

Evidence

Demo CNPG queries:

capacity_forecasts = 0 rows
ocsf_events = 51,036 total but 0 rows with source_type/event_type of anomaly | capacity_forecast
seasonal_profiles — relation "seasonal_profiles" does not exist
oban_jobs — no row whose worker matches forecast / capacity / anomaly
cpu_metrics = 231,112 and memory_metrics = 115,556 in the last 3d (source data present)

Code:

elixir/serviceradar_core/config/runtime.exs:1184-1187 — anomaly_analysis_enabled = System.get_env("ANOMALY_ANALYSIS_CONSUMER_ENABLED", "false") in ~w(true 1 yes) (defaults OFF)
elixir/serviceradar_core/lib/serviceradar/cluster/coordinator_children.ex:332-334 — anomaly_analysis_consumer_child only starts AnomalyDetection.Supervisor when AnomalyDetection.Config.enabled?()
elixir/serviceradar_core/config/runtime.exs:1058-1095 — capacity forecasting config; capacity_forecasting_crontab (runtime.exs:1084-1092) is appended to the Oban Cron plugin crontab at runtime.exs:1141 only when capacity_forecasting_enabled (default "true")

Demo deployment:

Core pod image sha-20def957 (pod serviceradar-core-55cb48d58c-f8gg5) env has SERVICERADAR_ANOMALY_* + SERVICERADAR_CAPACITY_FORECAST_CONFIG_* (config seed only) but no ANOMALY_ANALYSIS_CONSUMER_ENABLED and no SERVICERADAR_CAPACITY_FORECASTING_ENABLED.

This matches the dead verdict-spine reported in #3706 / #3710 / #3711 — the upstream producer is simply not switched on.

UI impact:

The finding tile reads 0/0/0.
The dashboard panel shows "No capacity forecast data" (web-ng DashboardPanelChart.jsx:519).

Recommendation

Before judging the UI, get the spine producing data:

Set ANOMALY_ANALYSIS_CONSUMER_ENABLED=true on the demo core pod, then verify AnomalyDetection.Supervisor starts and a verdict actually reaches ocsf_events.
Confirm SERVICERADAR_CAPACITY_FORECASTING_ENABLED is true and that the Oban crontab is actually loaded. The absence of ANY CapacityForecasting.Worker job despite the code default of "true" suggests the crontab plugin entry is gated out, or the worker enqueue is not happening — investigate the capacity_forecasting_crontab wiring (runtime.exs:1084-1092) vs the live Oban Cron plugin config (runtime.exs:1126-1142).
Create/refresh the seasonal_profiles table and its batch job so the forecaster has a seasonal baseline.

Until these are addressed, the tile will read 0/0/0 and the dashboard panel will keep showing "No capacity forecast data" regardless of how well the UI was built.

🤖 Generated with Claude Code

## Problem The causal anomaly-detection and capacity-forecast UI surfaces (tile, dashboard visual, event findings) render **empty in the live demo** because the data-production spine that should feed them is **not running at all** in the demo deployment. Every UI element that was built (Phases 4.4 + 5.1-5.4 of OpenSpec `add-causal-anomaly-detection`) has nothing to show: the tile reads 0/0/0 and the dashboard panel shows "No capacity forecast data". This is upstream of, and distinct from, the verdict-routing/alert-spine gaps in #3706 / #3710 / #3711 (which assume verdicts are being produced and then dropped in transit). Here, nothing is produced in the first place. ## Severity **blocker** — the entire feature produces zero data in the demo; the UI is empty and cannot be evaluated. ## Root cause The data-production spine is not running in the demo deployment: 1. **Real-time anomaly detector defaults OFF.** `ANOMALY_ANALYSIS_CONSUMER_ENABLED` defaults to `"false"` in `runtime.exs`, and it is **not set** on the demo core pod. The consumer (and `AnomalyDetection.Supervisor`) only starts when `AnomalyDetection.Config.enabled?()` is true, so the detector never runs. 2. **Capacity-forecasting Oban cron has never enqueued a single job in demo.** There are no `CapacityForecasting.Worker` rows in `oban_jobs` at all. The demo core pod has the config-seed env vars (`SERVICERADAR_CAPACITY_FORECAST_CONFIG_*`) but **not** `SERVICERADAR_CAPACITY_FORECASTING_ENABLED`. Even though the code default is `"true"`, the absence of ANY worker job suggests the crontab plugin entry is gated out or the enqueue is not happening — the crontab wiring needs verification against the loaded Oban plugin config. 3. **`seasonal_profiles` table does not even exist** (`relation "seasonal_profiles" does not exist`), so the seasonal baseline the forecaster depends on cannot be built. Source data IS present (`cpu_metrics`=231,112 / `memory_metrics`=115,556 in the last 3 days), so this is purely a "spine not running" problem, not a missing-input problem. ## Evidence Demo CNPG queries: - `capacity_forecasts` = **0 rows** - `ocsf_events` = 51,036 total but **0** rows with `source_type`/`event_type` of `anomaly` | `capacity_forecast` - `seasonal_profiles` — `relation "seasonal_profiles" does not exist` - `oban_jobs` — **no row** whose `worker` matches `forecast` / `capacity` / `anomaly` - `cpu_metrics` = 231,112 and `memory_metrics` = 115,556 in the last 3d (source data present) Code: - `elixir/serviceradar_core/config/runtime.exs:1184-1187` — `anomaly_analysis_enabled = System.get_env("ANOMALY_ANALYSIS_CONSUMER_ENABLED", "false") in ~w(true 1 yes)` (defaults OFF) - `elixir/serviceradar_core/lib/serviceradar/cluster/coordinator_children.ex:332-334` — `anomaly_analysis_consumer_child` only starts `AnomalyDetection.Supervisor` when `AnomalyDetection.Config.enabled?()` - `elixir/serviceradar_core/config/runtime.exs:1058-1095` — capacity forecasting config; `capacity_forecasting_crontab` (runtime.exs:1084-1092) is appended to the Oban Cron plugin crontab at runtime.exs:1141 only when `capacity_forecasting_enabled` (default `"true"`) Demo deployment: - Core pod image `sha-20def957` (pod `serviceradar-core-55cb48d58c-f8gg5`) env has `SERVICERADAR_ANOMALY_*` + `SERVICERADAR_CAPACITY_FORECAST_CONFIG_*` (config seed only) but **no** `ANOMALY_ANALYSIS_CONSUMER_ENABLED` and **no** `SERVICERADAR_CAPACITY_FORECASTING_ENABLED`. This matches the dead verdict-spine reported in #3706 / #3710 / #3711 — the upstream producer is simply not switched on. UI impact: - The finding tile reads 0/0/0. - The dashboard panel shows "No capacity forecast data" (`web-ng` `DashboardPanelChart.jsx:519`). ## Recommendation Before judging the UI, get the spine producing data: 1. Set `ANOMALY_ANALYSIS_CONSUMER_ENABLED=true` on the demo core pod, then verify `AnomalyDetection.Supervisor` starts and a verdict actually reaches `ocsf_events`. 2. Confirm `SERVICERADAR_CAPACITY_FORECASTING_ENABLED` is true **and** that the Oban crontab is actually loaded. The absence of ANY `CapacityForecasting.Worker` job despite the code default of `"true"` suggests the crontab plugin entry is gated out, or the worker enqueue is not happening — investigate the `capacity_forecasting_crontab` wiring (runtime.exs:1084-1092) vs the live Oban Cron plugin config (runtime.exs:1126-1142). 3. Create/refresh the `seasonal_profiles` table and its batch job so the forecaster has a seasonal baseline. Until these are addressed, the tile will read 0/0/0 and the dashboard panel will keep showing "No capacity forecast data" regardless of how well the UI was built. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

mfreeman451 referenced this issue

2026-06-13 06:18:30 +00:00

[anomaly][design] Device-details anomaly & capacity section — implementation design #3752

mfreeman451 referenced this issue

2026-06-13 09:11:15 +00:00

[anomaly-detection + capacity-forecast UI: state and surfacing (OpenSpec add-causal-anomaly-detection, Phases 4.4 + 5.1-5.4)][recommendation] UI was specced (4.4, 5.1-5.4) and is built + merged to staging — reuse, don't rebuild #3730

mfreeman451 referenced this issue

2026-06-13 11:08:46 +00:00

fix: enable causal anomaly spine in demo #3776

mfreeman451 commented

2026-06-13 11:08:59 +00:00

Author

Owner

Fix PR opened: #3776 (work/3731-enable-causal-spine-demo).

What changed:

Added explicit Helm values for core.anomalyAnalysis and core.capacityForecasting.
Rendered ANOMALY_ANALYSIS_CONSUMER_ENABLED=true in values-demo.yaml so the real-time anomaly consumer actually starts in demo after rollout.
Rendered anomaly NATS URL/creds/TLS env (ANOMALY_ANALYSIS_NATS_*) so enabling the consumer does not fall back to localhost/defaults.
Rendered SERVICERADAR_CAPACITY_FORECASTING_* worker envs so Oban cron/runtime forecasting is explicitly enabled, not just the first-boot config seeder.
Fixed Helm rendering for forecast horizon/min-point values so they render as integer strings (7776000, 2592000, 72) instead of scientific notation, which would break runtime integer parsing.

Validation:

helm template serviceradar ./helm/serviceradar -n demo -f helm/serviceradar/values-demo.yaml
helm template serviceradar ./helm/serviceradar -n default
helm lint ./helm/serviceradar -f helm/serviceradar/values-demo.yaml
git diff --check

Important note: current staging capacity forecasting does not use a seasonal_profiles table; it reads SRQL hourly metric histories directly and writes platform.capacity_forecasts. Leaving this issue open until #3776 lands, demo is rolled to the new build/chart, and we verify capacity_forecasts plus anomaly/capacity OCSF rows are non-empty in demo.

Fix PR opened: #3776 (`work/3731-enable-causal-spine-demo`). What changed: - Added explicit Helm values for `core.anomalyAnalysis` and `core.capacityForecasting`. - Rendered `ANOMALY_ANALYSIS_CONSUMER_ENABLED=true` in `values-demo.yaml` so the real-time anomaly consumer actually starts in demo after rollout. - Rendered anomaly NATS URL/creds/TLS env (`ANOMALY_ANALYSIS_NATS_*`) so enabling the consumer does not fall back to localhost/defaults. - Rendered `SERVICERADAR_CAPACITY_FORECASTING_*` worker envs so Oban cron/runtime forecasting is explicitly enabled, not just the first-boot config seeder. - Fixed Helm rendering for forecast horizon/min-point values so they render as integer strings (`7776000`, `2592000`, `72`) instead of scientific notation, which would break runtime integer parsing. Validation: - `helm template serviceradar ./helm/serviceradar -n demo -f helm/serviceradar/values-demo.yaml` - `helm template serviceradar ./helm/serviceradar -n default` - `helm lint ./helm/serviceradar -f helm/serviceradar/values-demo.yaml` - `git diff --check` Important note: current staging capacity forecasting does not use a `seasonal_profiles` table; it reads SRQL hourly metric histories directly and writes `platform.capacity_forecasts`. Leaving this issue open until #3776 lands, demo is rolled to the new build/chart, and we verify `capacity_forecasts` plus anomaly/capacity OCSF rows are non-empty in demo.

mfreeman451 commented

2026-06-13 11:38:44 +00:00

Author

Owner

PR #3776 is now merged into staging.

What is fixed in code/chart:

demo renders ANOMALY_ANALYSIS_CONSUMER_ENABLED=true and the required anomaly NATS URL/creds/TLS envs
capacity forecasting worker envs are explicit in Helm values/templates
forecast numeric envs render as integer strings instead of scientific notation

Leaving this issue open until the next demo rollout/e2e pass proves the live behavior changed: platform.capacity_forecasts should become non-empty, and anomaly/capacity OCSF output should appear in demo. That validation belongs to the demo rollout before the v1.3.1 release.

PR #3776 is now merged into `staging`. What is fixed in code/chart: - demo renders `ANOMALY_ANALYSIS_CONSUMER_ENABLED=true` and the required anomaly NATS URL/creds/TLS envs - capacity forecasting worker envs are explicit in Helm values/templates - forecast numeric envs render as integer strings instead of scientific notation Leaving this issue open until the next demo rollout/e2e pass proves the live behavior changed: `platform.capacity_forecasts` should become non-empty, and anomaly/capacity OCSF output should appear in demo. That validation belongs to the demo rollout before the v1.3.1 release.

mfreeman451 referenced this issue

2026-06-13 12:32:30 +00:00

fix: enable causal anomaly spine in demo #3782

mfreeman451 commented

2026-06-13 12:32:42 +00:00

Author

Owner

Opened PR #3782 to address the demo producer-spine gap.

What changed:

Helm now exposes the anomaly analysis consumer envs on core-elx, including ANOMALY_ANALYSIS_CONSUMER_ENABLED, NATS URL, creds file, and TLS toggle.
Demo values set anomalyAnalysis.enabled=true, so AnomalyDetection.Supervisor can start in demo.
Helm now exposes the capacity forecasting runtime envs used by runtime.exs and the Oban cron wiring.
Demo values set capacityForecasting.enabled=true and emitVerdicts=true.

Validation performed on the exact PR commit:

helm lint helm/serviceradar -f helm/serviceradar/values-demo.yaml
helm template serviceradar helm/serviceradar -n demo -f helm/serviceradar/values-demo.yaml --show-only templates/core.yaml, confirming ANOMALY_ANALYSIS_CONSUMER_ENABLED=true and SERVICERADAR_CAPACITY_FORECASTING_ENABLED=true in the rendered demo core deployment.

One correction to the original issue: the seasonal_profiles table finding is stale against current code. The current forecaster stores projections in platform.capacity_forecasts and applies seasonal modeling in ServiceRadar.Observability.CapacityForecasting.Model; there are no seasonal_profiles table references in current core code.

Leaving this issue open until #3782 is merged and the demo rollout confirms capacity_forecasts / anomaly verdict production.

Opened PR #3782 to address the demo producer-spine gap. What changed: - Helm now exposes the anomaly analysis consumer envs on core-elx, including ANOMALY_ANALYSIS_CONSUMER_ENABLED, NATS URL, creds file, and TLS toggle. - Demo values set anomalyAnalysis.enabled=true, so AnomalyDetection.Supervisor can start in demo. - Helm now exposes the capacity forecasting runtime envs used by runtime.exs and the Oban cron wiring. - Demo values set capacityForecasting.enabled=true and emitVerdicts=true. Validation performed on the exact PR commit: - helm lint helm/serviceradar -f helm/serviceradar/values-demo.yaml - helm template serviceradar helm/serviceradar -n demo -f helm/serviceradar/values-demo.yaml --show-only templates/core.yaml, confirming ANOMALY_ANALYSIS_CONSUMER_ENABLED=true and SERVICERADAR_CAPACITY_FORECASTING_ENABLED=true in the rendered demo core deployment. One correction to the original issue: the seasonal_profiles table finding is stale against current code. The current forecaster stores projections in platform.capacity_forecasts and applies seasonal modeling in ServiceRadar.Observability.CapacityForecasting.Model; there are no seasonal_profiles table references in current core code. Leaving this issue open until #3782 is merged and the demo rollout confirms capacity_forecasts / anomaly verdict production.

mfreeman451 referenced this issue

2026-06-13 12:36:40 +00:00

fix: enable causal anomaly spine in demo #3782

mfreeman451 commented

2026-06-13 12:36:49 +00:00

Author

Owner

Correction: #3782 was a duplicate/no-op PR and I closed it. The actual #3731 Helm/runtime fix is already merged as PR #3776.

Current evidence:

origin/staging already contains the demo values for anomalyAnalysis.enabled=true and capacityForecasting.enabled=true.
work/3731-enable-causal-spine-demo has zero diff against origin/staging.
PR #3776 rendered/linted the chart and documents the same seasonal_profiles correction: the current forecaster writes platform.capacity_forecasts and uses ServiceRadar.Observability.CapacityForecasting.Model, not a seasonal_profiles table.

Remaining close condition for this issue is runtime validation in demo after rollout: capacity_forecasts should become non-empty and anomaly/capacity verdict rows should appear where live input triggers them.

Correction: #3782 was a duplicate/no-op PR and I closed it. The actual #3731 Helm/runtime fix is already merged as PR #3776. Current evidence: - origin/staging already contains the demo values for anomalyAnalysis.enabled=true and capacityForecasting.enabled=true. - work/3731-enable-causal-spine-demo has zero diff against origin/staging. - PR #3776 rendered/linted the chart and documents the same seasonal_profiles correction: the current forecaster writes platform.capacity_forecasts and uses ServiceRadar.Observability.CapacityForecasting.Model, not a seasonal_profiles table. Remaining close condition for this issue is runtime validation in demo after rollout: capacity_forecasts should become non-empty and anomaly/capacity verdict rows should appear where live input triggers them.

mfreeman451 referenced this issue

2026-06-13 14:45:46 +00:00

Handoff: resume demo rollout and E2E from staging 98b2f55 #3786