Plan longer-horizon forecasting and pgvector-backed similarity search #3811

New issue

Open

opened 2026-06-14 00:47:41 +00:00 by mfreeman451 · 0 comments

mfreeman451 commented

2026-06-14 00:47:41 +00:00

Owner

Why

ServiceRadar needs longer-term forecasting and capacity/trend intelligence beyond the hot anomaly detector. pgvector is available in CNPG and is useful for offline/nearline similarity, clustering, embedding search, and explanation retrieval, but it must not be used for per-sample detector hot state. Any DB read/write in the evaluation loop will crush throughput.

Scope

Design a long-horizon forecasting path for capacity planning and trend analysis.
Evaluate pgvector for grouping similar series behavior, embedding forecast windows, anomaly explanation retrieval, and capacity/trend feature search.
Define how forecast windows/features are produced from existing metric history without bypassing JetStream ingestion.
Keep hot detector state shard-local in Rust using Welford/ring counters.
Checkpoint compact detector snapshots asynchronously to object storage or CNPG outside the evaluation loop.
Query CNPG/pgvector only outside the per-sample evaluation path.

Non-goals / guardrails

Do not put per-sample anomaly detector state in pgvector or any synchronous DB-backed store.
Do not add DB reads/writes to the anomaly evaluation loop.
Do not create metric paths that bypass NATS JetStream/event_writer.

Initial implementation questions

What feature vector represents a series/window for similarity search?
What retention/downsampling levels are needed for 30/90/180-day forecasts?
Which jobs build embeddings/features, and how are they scheduled/backfilled?
How do forecast results feed health pages, device pages, and capacity runway UI without coupling to hot detection?
What checkpoint format is needed for native shard detector recovery, and where should snapshots live?

Acceptance criteria for the follow-up proposal

OpenSpec proposal separates hot detector state, async checkpointing, and offline/nearline forecast/search paths.
Benchmarks or sizing estimates show DB/vector search is not in the per-sample path.
Data model uses platform schema migrations only and keeps all metrics sourced through JetStream/event_writer.

## Why ServiceRadar needs longer-term forecasting and capacity/trend intelligence beyond the hot anomaly detector. pgvector is available in CNPG and is useful for offline/nearline similarity, clustering, embedding search, and explanation retrieval, but it must not be used for per-sample detector hot state. Any DB read/write in the evaluation loop will crush throughput. ## Scope - Design a long-horizon forecasting path for capacity planning and trend analysis. - Evaluate pgvector for grouping similar series behavior, embedding forecast windows, anomaly explanation retrieval, and capacity/trend feature search. - Define how forecast windows/features are produced from existing metric history without bypassing JetStream ingestion. - Keep hot detector state shard-local in Rust using Welford/ring counters. - Checkpoint compact detector snapshots asynchronously to object storage or CNPG outside the evaluation loop. - Query CNPG/pgvector only outside the per-sample evaluation path. ## Non-goals / guardrails - Do not put per-sample anomaly detector state in pgvector or any synchronous DB-backed store. - Do not add DB reads/writes to the anomaly evaluation loop. - Do not create metric paths that bypass NATS JetStream/event_writer. ## Initial implementation questions - What feature vector represents a series/window for similarity search? - What retention/downsampling levels are needed for 30/90/180-day forecasts? - Which jobs build embeddings/features, and how are they scheduled/backfilled? - How do forecast results feed health pages, device pages, and capacity runway UI without coupling to hot detection? - What checkpoint format is needed for native shard detector recovery, and where should snapshots live? ## Acceptance criteria for the follow-up proposal - OpenSpec proposal separates hot detector state, async checkpointing, and offline/nearline forecast/search paths. - Benchmarks or sizing estimates show DB/vector search is not in the per-sample path. - Data model uses platform schema migrations only and keeps all metrics sourced through JetStream/event_writer.

mfreeman451 referenced this issue

2026-06-14 01:02:30 +00:00

impl: add DeepCausality compact reasoner state #3799