Unified metric ingestion contract across agent/sysmon/snmp, native add-ons, and wasm plugins (OCSF-correct) #3788

New issue

Open

opened 2026-06-13 19:41:52 +00:00 by mfreeman451 · 5 comments

mfreeman451 commented 2026-06-13 19:41:52 +00:00

Owner

Copy link

TL;DR

A single CPU-utilization metric reaches the anomaly engine through three structurally disjoint pipelines (Go agent sysmon, WASM/native plugin via plugin_result, native add-on via OTLP relay) that converge only on shared gRPC transport (AgentGatewayService.StreamStatus with opaque JSON in GatewayServiceStatus.Message, disambiguated by an agent-set Source string) and shared IngressId/Sr-Ingress-* attribution headers — never on schema, subject, point-identity, or storage. Three series-identity algorithms and three DB sinks (cpu_metrics, timeseries_metrics, otel_metric_points) mean the same physical core produces three uncorrelatable anomaly series, and only the OTLP path is analyzed by default.

The core defects, all verified against code:

1. Metrics are smuggled inside status/check-result envelopes because producers have no first-class metric emit path. The agent-gateway then does a parse-forward-reparse-republish dance — the same metric JSON is decoded up to three times (gateway publisher → core StatusHandler after forward → core Metrics processor off NATS).
2. Metric-shape fragmentation is the real disease, not OCSF: 4 wire schemas, 3 sinks, 3 non-alignable series-identity recipes, no shared point identity.
3. No kind / temporality anywhere in the producer SDKs, and the anomaly consumer discards kind/temporality/unit even when the OTLP path supplies them — a cumulative counter fed raw into z-sigma looks like a ramp.

OCSF is NOT mis-used. ServiceRadar correctly keeps metric time-series out of OCSF classes; the separation is deliberate and documented (elixir/serviceradar_core/lib/serviceradar/event_writer/ocsf.ex:16-20). The fix is metric-shape unification, which is orthogonal to OCSF and must not be solved by routing into it.

The fix is additive, not a rewrite. PR #3787 already shipped serviceradar.metric.v1 as a thin bridge and is the de-facto reference implementation. Evolve it to an OTLP-grade contract (schema_version 1→2, add resource/kind/temporality/points[]), give all producers a first-class metric emit path, lower the legacy sysmon/snmp/shadow special-cases onto it, and delete the migration-era debt.

---

Architectural principle

Metrics are a first-class signal. Producers (WASM plugins, native add-ons, and the agent sysmon/snmp/icmp/mtr/sweep/rperf collectors) emit one serviceradar.metric.v1 shape directly onto a metric path — never embedded inside a status / check-result / plugin-result envelope. The gateway stops re-extracting metrics from statuses, and the per-source (sysmon/snmp/plugin) special-cases collapse into one emit + one ingest + one sink.

---

Current state — four disjoint pipelines (CPU metric traced end-to-end)

	Go agent sysmon	WASM/native plugin	native add-on OTLP relay
producer shape	MetricSample.cpus[]{core_id,usage_percent} (go/pkg/sysmon/metrics.go:25-180)	Metric{name,value,unit,warn,crit,min,max} (serviceradar-sdk-go/sdk/result.go:52-60; Rust src/result.rs:138-184)	OTLP NumberDataPoint in ExportMetricsServiceRequest
wrap / source	JSON in GatewayServiceStatus.Message, Source="sysmon-metrics" (push_loop_status.go:200-233)	embedded in serviceradar.plugin_result.v1 over env.submit_result, Source="plugin-result"	opaque OTLP, Source="otlp-relay"
gateway publisher	SysmonMetricsPublisher → serviceradar.sysmon.metrics.v1 → metrics.sysmon.<family>	PluginMetricsPublisher → serviceradar.metric.v1 → metrics.timeseries.<type>.<name>	OtlpRelayPublisher → verbatim → otel.metrics.raw
stream	metrics	metrics	events (separate)
core processor → sink	Metrics→SysmonMetricsIngestor → cpu_metrics per-family hypertable	Metrics→Telemetry → timeseries_metrics	OtelMetrics (protobuf) → otel_metric_points
anomaly series id	sysmon:cpu:<host>:<core_id>	cpu:<TimeseriesSeriesKey md5>	otel:<svc>:<name>:<attributes_hash>
kind/temporality/unit	none	none (unit stored, then dropped)	full — then discarded by consumer

The gateway is not a normalizer — it is four source-keyed pass-through publishers selected by the agent-supplied Source string (status_processor.ex:57-97,328-431).

Divergences (why the same core yields three uncorrelatable series)

• 3 series-identity algorithms, none alignable: sysmon string-built (sample_extractor.ex ~:123), plugin TimeseriesSeriesKey md5 (timeseries_series_key.ex:6-35), OTLP attributes_hash recipe-v2 (otel_metric_point.ex:19-198).
• 3 DB sinks / schemas with no unifying view or join (repo-wide search found no UNION/view merging timeseries_metrics and otel_metric_points).
• 3 wire schemas / encodings: serviceradar.sysmon.metrics.v1 (whole-sample blob), serviceradar.metric.v1 (per-scalar JSON), raw OTLP protobuf.
• Coverage asymmetry: anomaly enabled_subjects defaults to ['otel.metrics.>'] only (anomaly_detection/config.ex:18), so only the native-addon OTLP CPU metric is analyzed out of the box; sysmon and plugin CPU are subscribed but gated off until ANOMALY_ANALYSIS_ENABLED_SUBJECTS is widened.
• Per-device identity collapse on the plugin path only: serviceradar.metric.v1 never sets device_id/target_device_ip/if_index and per-metric labels are not promoted to tags, so many devices under one agent collide into one series (telemetry.ex:112-137).
• Timestamp semantics differ: OTLP has true per-point time_unix_nano; sysmon shares one sample-level timestamp across all cores; plugin shares the Result envelope observed_at across all metrics.
• Consumer destroys fidelity uniformly: SampleExtractor → ContextOwner → CausalReasoner reduces every path to {value, observed_at_unix_nano} (sample_extractor.ex, context_owner.ex:349-363). kind/temporality/unit/is_monotonic never reach the reasoner even when otel_metric_points has them.

---

The double/triple-decode (the "re-publish" smell)

status_processor.ex forward_then_publish/1 (:78-97) does two passes over the same payload: (1) forward(status) ships the whole status to core for status/state processing, then (2) re-parses status.message and republishes the extracted metrics onto JetStream via three source-specific publishers. Evidence of the same metric JSON decoded three times:

• sysmon_metrics_publisher.ex:42-51 / snmp_metrics_publisher.ex:44-53 / plugin_metrics_publisher.ex:44-53 — each Jason.decode(status.message) (decode #2).
• metrics.ex:46-72 — core re-decodes the republished NATS message (decode #3).

It is not a duplicate DB write today: the direct-insert sink is flag-gated off (plugin_result_ingestor.ex:6-10,:52,:200-206, :plugin_result_direct_metrics_enabled default false), and the core gRPC-direct-to-DB metric handlers are already no-op stubs (results_router.ex:239-245,:611 → acknowledge_metrics_cutover/1 → :ok). So half the migration is already done — it just hasn't been cleaned up. The remaining smell is purely the smuggle-and-re-extract architecture, which the first-class metric path eliminates.

---

OCSF finding — NOT a category error (keep the separation, add guardrails)

This was investigated directly and the "metrics are mis-bucketed into OCSF" claim was REFUTED across producers, transport, consumers, and schema:

• Policy is documented in code: ocsf.ex:16-20 — "OTel traces/metrics: Keep native format (observability, not security)" and "Telemetry metrics: Keep native format (time-series data)".
• ocsf_events (migration 20260203120000_create_ocsf_events.exs:14-45) is a pure security-event shape (class_uid/category_uid/activity_id/severity_id/observables/actor/device/endpoints) with zero metric value/unit/temporality/metric_name columns — it physically cannot hold a series.
• Both WASM SDKs only have ocsf_event/otel_log payload kinds (serviceradar-sdk-go/sdk/signal_schema.go:17-18, serviceradar-sdk-rust/src/result.rs:356-357) — there is no metric signal type, so a metric cannot be serialized as an OCSF event.
• The only numbers legitimately inside OCSF are (a) NetFlow byte/packet counters as traffic{} in ocsf_network_activity class 4001 (event-shaped, OCSF-sanctioned), and (b) anomaly/capacity Findings (class 2004) carrying a single anomalous value + thresholds as event context (verdict_emitter.ex:34-74,:96-118) — derived from metrics, not duplicating the series. Both are correct and stay.

Recommendations:

• Do NOT add a "metric" OCSF class. OCSF has no metric/time-series primitive; inventing one would be the actual category error.
• Keep using OCSF for events derived from metrics (threshold breaches → Findings) — the code already does this correctly.
• Add mis-bucket guardrails with the unified contract: the metrics processor rejects any serviceradar.metric.v1 payload carrying class_uid; the OCSF processor rejects payloads carrying kind/temporality/points[]. A misrouted message then errors loudly instead of landing in the wrong hypertable.

---

Confirmed gaps (verified against code)

1. No metric kind (gauge/counter/histogram) in any producer SDK or in three of four gateway envelopes — result.go:52-60, result.rs:138-184. Downstream metric_type is a routing label, not an OTel kind.
2. No temporality (delta/cumulative) in SDKs or the serviceradar.metric.v1 / serviceradar.sysmon.metrics.v1 envelopes (plugin_metrics_publisher.ex:93-111). SNMP carries only is_delta. Result: monotonic counters look like ramps to z-sigma.
3. No first-class metric emit helper in either WASM SDK — telemetry.go has only NewOCSFTelemetryRecord(:36) and NewOTELLogTelemetryRecord(:53); telemetry.rs only ocsf_event(:60)/otel_log(:76). Metrics can only ride embedded in submit_result; there is no metric payload kind.
4. No per-metric tags / resource identity / point identity on the plugin_result Metric shape — labels are envelope-level (result.go:37, result.rs:385); per-metric dimensions are lost (plugin_metrics_publisher.ex:113-191).
5. Per-device identity collapse on the plugin path — serviceradar.metric.v1 never sets device_id/target_device_ip/if_index, so many devices under one agent collapse into one series.
6. metrics.timeseries.* and otel.metrics.raw decode into different sinks (timeseries_metrics vs otel_metric_points) with no canonical merge.
7. Anomaly consumer drops kind/temporality/unit at extraction even when OTLP supplies them (sample_extractor.ex, context_owner.ex:349-363).
8. Coverage losses inside the (correct) metric namespace: OTLP exponential histograms & summaries are counted-then-dropped (otel_metrics.ex ~:258-281); the SNMP gateway publisher drops every OID except ifHCInOctets/ifHCOutOctets (snmp_metrics_publisher.ex); sysmon Network[] interface metrics are collected but dropped at the gateway (only cpu/memory/disk/process families emitted).
9. No Nats-Msg-Id / duplicate_window on the metrics stream (config.ex:206-219). With consumer_max_deliver=5 and the planned multi-point fan-out, at-least-once delivery would double-count; idempotency must be enforced at the stream and/or ingestor.
10. No schema/proto validation of serviceradar.metric.v1 — the producer hand-builds the map and the consumer tolerantly defaults missing fields (metric_type→'gauge'); malformed envelopes become gauge/unknown rows instead of being rejected. Non-numeric metrics are silently dropped (encode_metric returns {:ok, nil} with no counter — invisible data loss).

---

Producer refactor work-list (every producer smuggling metrics through status)

Swept all 12 native add-ons, both WASM SDKs, and in-tree first-party checkers (22 producers classified).

SMUGGLES_METRICS — first-party in-tree checkers (the real refactor targets)

producer	metrics	source string	evidence
sysmon	cpu/disk/memory/network-iface/process	sysmon-metrics	go/pkg/sysmon/metrics.go:25-180, push_loop_status.go:200-233
snmp	OID counters/gauges w/ if_index,interface_uid	snmp-metrics	push_loop_snmp.go:30-42,48-57,63
icmp	response_time_ns, packet_loss	results	push_loop_icmp_results.go:29-68, icmp_checker.go:69-77
mtr	per-hop RTT/loss/jitter	results	mtr_checker.go:61-70,86-127, proto/monitoring.proto:1012-1046
network sweep	scanner perf counters (SweepScannerStats)	results	proto/monitoring.proto:149-200, push_loop_sweep_results.go:28-118
mapper/discovery	per-interface metric descriptors (InterfaceMetric) — catalog, not samples	discovery status	proto/discovery/discovery.proto:188-214, push_loop_mapper_netprobe.go:36-93
rperf (Rust checker)	throughput/loss/jitter (TestSummary)	plugin-result (generic path)	proto/rperf/rperf.proto:60-72, rust/rperf-client/src/server.rs:259-300

Note: mapper/discovery carries metric definitions (the catalog the snmp checker samples), not time-series — decide whether the unified contract carries metric-descriptors on a separate channel.

SDK smuggling helpers (add first-class emit here, then deprecate)

• Go: Result.Metrics slot (serviceradar-sdk-go/sdk/result.go:36), Metric struct (:52-60), AddMetric/WithMetric builders (:257-273).
• Rust: Metric struct (serviceradar-sdk-rust/src/result.rs:138-151), add_metric_spec/with_metric_spec (:578-585), add_metric/with_metric (:587-612).
• 6 SDK examples (http-check, udp-check, tcp-check, widgets-check ×Go; http-check, widgets-check ×Rust) and 2 golden fixtures (testdata/service_monitoring_result.json ×2) demonstrate the pattern and must update in lockstep.

OTLP_NATIVE (already first-class)

• otel-addon + rust/otel library relay customer OTLP metrics via the acked RelayOtlp stream (otlp-relay:v1): rust/otel-addon/src/addon.rs:27,556, rust/otel/src/lib.rs:529,754, addon.proto:159-165 (OTLP_METRICS=5, OTLP_DERIVED_METRIC=6). Decision needed: normalize OTLP_METRICS frames into serviceradar.metric.v1 downstream, or keep otel.metrics.raw as a high-fidelity express lane.

NO_METRICS (no refactor) — explicitly verified to avoid false positives

bumblebee-scan, endpoint-inventory, scalibr-endpoint-inventory, advisory-producer, sample-addon, powerdns, rdp-adapter, workload-identity, bmp-collector, log-collector, trapd, flowgger, rust-sample-addon, rperf-server. Correction to parallel sub-agents: netprobe, flow-collector, and otel-addon's StreamTelemetry expose only operational Prometheus/OCSF-usage self-metrics, NOT smuggled observability time-series — they are NOT refactor targets.

---

Tech-debt to delete / collapse (the unification makes the special-cases deletable)

DELETE (pure migration-era scaffolding)

• @legacy_sysmon_schema "serviceradar.sysmon.shadow.v1" acceptance — metrics.ex:19,:151.
• AGENT_GATEWAY_SYSMON/SNMP_METRICS_SHADOW_ENABLED env defaults (runtime.exs:314,318, default "true") + helm *MetricsShadowEnabled aliases (agent-gateway.yaml:30,36).
• Flag-gated legacy direct insert maybe_insert_metrics + :plugin_result_direct_metrics_enabled (plugin_result_ingestor.ex:52,:200-206; core/config.exs:131-133). Keep build_status_row/insert_status — only the metric insert is debt.
• Vestigial no-op router handlers handle_sysmon_metrics/handle_snmp_metrics/acknowledge_metrics_cutover (results_router.ex:239-245,:611) and their dispatch entries (status_handler.ex:109-111).

COLLAPSE (replace source-specific handling with the unified path)

• SysmonMetricsPublisher (sysmon_metrics_publisher.ex whole module) and SnmpMetricsPublisher (snmp_metrics_publisher.ex whole module) → fold into a generalization of PluginMetricsPublisher (which already emits serviceradar.metric.v1). These are the publish boundary (agents are NATS-denied), so they must be replaced, not merely deleted.
• Sysmon family-envelope carve-out in metrics.ex:18,50,59-181 (parse_sysmon/filter_sysmon_sample/partition_messages/ingest_sysmon_messages/metric_kind) → the generic Telemetry/timeseries path.
• SysmonMetricsIngestor (sysmon_metrics_ingestor.ex) 5-hypertable fan-out → unified ingest.
• Gateway double-pass forward_then_publish + publish_sysmon/snmp/plugin_metrics + *_source? guards (status_processor.ex:78-97,313-427); per-source max_message_bytes/strict_message_size_source? (agent_gateway_server.ex:665-680) — preserve the large sysmon size budget on the unified path.
• Anomaly per-source carve-outs: sample_extractor.ex:67-71,114-189, series_config.ex:201-214, config.ex:108-109 (ANALYSIS_METRICS_SYSMON/SNMP).
• Per-source subject prefixes / NATS ACLs (config.exs:30,35, runtime.exs:346,351, agent-gateway.yaml:127,131, nats.yaml:211, values-demo.yaml:488, values.yaml:946,948) — re-bind stream subjects + durable consumer filters atomically during cutover.

MIGRATE (producer code switches to first-class emit)

• Go agent sysmon producer (push_loop_status.go:118-233, sysmon_service.go:604-612, go/pkg/sysmon/{metrics,collector}.go) — flatten cpus/clusters/disks/processes into per-series scalar points.
• Go agent SNMP producer (push_loop_snmp.go:66,77,101,140) — keep if_index/target_device_ip identity.
• (+ icmp/mtr/sweep/rperf per the producer work-list above.)

KEEP — do NOT delete (sysmon-named but config/UI, not metric debt)

• sysmon_profiles table + migrations (20260519162000_*, 20260123093000_*), sysmon_compiler.ex, settings UI (sysmon_profiles_live/index.ex) — configuration.
• Device-detail sysmon charts UI (device_tab_runtime.ex) and SRQL catalog — repoint to the unified table, do not delete.
• go/pkg/sysmon/collector.go + sysmon_service.go — the collector itself (migrate the emit shape, not the collector).
• The five sysmon metric hypertables (cpu_metrics/cpu_cluster_metrics/memory_metrics/disk_metrics/process_metrics) + _hourly caggs — unification candidates, but they back live UI charts and anomaly baselines: data-migrate before any DROP.

Tests that lock in the per-source schemas/subjects (will need rewriting)

metrics_test.exs (legacy shadow + sysmon/snmp fixtures), sysmon_metrics_publisher_test.exs, snmp_metrics_publisher_test.exs, status_processor_test.exs:428,456, sysmon_metrics_ingestor_test.exs, snmp_metrics_ingestor_integration_test.exs, results_router_test.exs:431,445, sample_extractor_test.exs, pipeline_test.exs, config_test.exs:40-88, series_config_test.exs/synthetic_dataset_test.exs/verdict_emitter_test.exs/baseline_seeder_test.exs/stateful_alert_engine_test.exs, jetstream_consumer_test.exs, pipeline_ack_test.exs/pipeline_batch_span_test.exs.

---

Proposed unified contract: serviceradar.metric.v1 (evolve additively from the PR #3787 bridge)

Reuse the existing schema id; bump schema_version 1→2 and add the OTLP-grade fields below. Version-1 flat envelopes remain valid (treated as a single gauge point: value→points[0].value, metric_type→kind, tags→points[0].attributes, observed_at→points[0].time_unix_nano). Consumers MUST reject envelopes whose schema is absent/unknown rather than coercing to gauge.

field	type	purpose	OTLP equiv
schema	const "serviceradar.metric.v1"	route + version-gate	Schema URL analog
schema_version	uint32 (default 1)	additive evolution	—
resource	{service_name(req), service_instance_id, scope_name, scope_version, attributes:map}	resource identity carried once per batch/point — the field the SDK Metric and PR3787 envelope both lack. sysmon: service_name='sysmon', attrs carry host_id/core_id/mount_point/pid; snmp: target_device_ip/if_index/interface_uid	Resource + InstrumentationScope
name	string (req)	metric stream name (dotted semconv), no longer the sole identity	Metric.name
kind	enum {gauge, sum, histogram, exp_histogram, summary}	the field every producer is missing; lets the engine rate-compute counters	Metric.data oneof
temporality	enum {unspecified, delta, cumulative} (req for sum/histogram)	counter-rate computability; snmp is_delta→delta	aggregation_temporality
is_monotonic	bool (kind=sum)	counter vs UpDownCounter	Sum.is_monotonic
unit	string (UCUM: %,By,1,ms,Hz)	stops n_sigma mixing percent/bytes/counts	Metric.unit
points	array (req, ≥1)	one OR MANY points per emit so cores/mounts/interfaces/guests dimension without splitting envelopes (fixes per-device collapse)	data_points (repeated)
points[].time_unix_nano	uint64 (req)	per-point measurement time	DataPoint.time_unix_nano
points[].start_time_unix_nano	uint64 (req for cumulative rate)	counter reset/window anchor	DataPoint.start_time_unix_nano
points[].attributes	map<string,scalar>	per-point dims (core_id,mount_point,if_index,guest_id); volatile keys excluded from identity	DataPoint.attributes
points[].value	double|int (gauge/sum)	int vs double preserved	NumberDataPoint.as_double/as_int
points[].histogram	{count,sum,bucket_counts[],explicit_bounds[],min,max} (histogram)	first-class histogram (none today); enables p50/p95 series currently dropped	HistogramDataPoint
points[].exemplars	[{time_unix_nano,value,trace_id,span_id,filtered_attributes}] (opt)	metric→trace pivots the causal engine wants	DataPoint.exemplars
thresholds	{warn,crit,min,max} (opt)	preserves Nagios perfdata semantics without conflating with metric data; alerting/display only	— (deliberately outside OTLP)
ingress_id	UUIDv8 (gateway-stamped)	per-message ordering/idempotency; also set as Nats-Msg-Id	—
ingress_timestamp_unix_nano	uint64 (gateway-stamped)	ingest/transport time, distinct from measurement time	—
ingest_identity	{ingest_agent_id, ingest_partition, ingest_identity:'agent:<id>'} (gateway-attested)	mTLS-bound attested identity, separate from producer-claimed resource.attributes (untrusted)	attested overlay

Per-SDK emit APIs (new third signal stream: signal_type:'metric' / payload_kind:'otel_metric')

• go-wasm: sdk.NewGauge/NewCounter/NewHistogram(name, value, ...MetricOption) with WithUnit/WithAttributes/WithTimestamp/WithResource/WithThresholds/WithExemplar; collected into sdk.NewMetricBatch(resource); add NewMetricTelemetryRecord(batch) so metrics ride env.emit_telemetry. Keep Result.AddMetric as a shim lowering to a gauge point.
• rust-wasm: MetricPoint::gauge/counter/histogram(...) + MetricBatch::new(ResourceRef::service(...)) + TelemetryRecord::otel_metric(batch). Keep Metric/add_metric_spec as a perfdata shim.
• native add-on (rust/addon-sdk): MetricBatchBuilder::new(resource).gauge/counter/histogram(...) + Addon::emit_metrics(batch) wrapping a new TelemetryPayloadKind::SR_METRIC_V1 (alongside OTLP_METRICS=5, OTLP_DERIVED_METRIC=6). OTLP-native authors keep relay_otlp() unchanged.
• agent sysmon/snmp/icmp/mtr/sweep: internal go/pkg/metricpoint builder mirroring the SDK so the gateway publishers emit serviceradar.metric.v1 multi-point envelopes (one Metric{kind,unit,points[]} per family), replacing the raw-sample blob and the 2-OID hardcode; emit the currently-dropped sysmon Network[] and SNMP non-octet OIDs.
• shared: ResourceRef.service(name).instance(id).scope(name,ver).attr(k,v) so all producers construct the resource tuple once.

Subject topology

One canonical tree metrics.v1.<kind>.<domain>.<name-token> on the existing high-rate metrics stream (widen the binding to also own metrics.v1.>). Leading token = the real kind field (no longer regex-guessed). Legacy subjects (metrics.sysmon.*, metrics.snmp.interface.*, metrics.timeseries.<type>.<name>) stay bound during migration. The addon raw-OTLP path is NOT moved by alias — a decode-and-republish shim lowers ExportMetricsServiceRequest into metrics.v1.* for a single consumer shape (or keep otel.metrics.raw as the express lane — see open questions). Critical hardening before fan-out: gateway sets Nats-Msg-Id=ingress_id on every publish AND the metrics stream gains a duplicate_window. Add metrics.v1.> to ANOMALY_ANALYSIS_ENABLED_SUBJECTS.

---

Relationship to PR #3787

PR #3787 is a correct, deliberately-thin bridge that should be evolved, not replaced. It ships serviceradar.metric.v1 as a flat {schema, metric_name, metric_type, value, unit, tags, metadata} envelope on metrics.timeseries.<type>.<name> → Telemetry → timeseries_metrics, wires the anomaly consumer (ANALYSIS_METRICS_TIMESERIES), and cleanly gates off the legacy direct-DB insert. It is the de-facto reference implementation of the unified contract. It explicitly does not solve the resource-metric SDK contract: no kind/temporality, no resource identity (device_id/if_index/target_device_ip always nil → per-device collapse), no per-metric labels, and metric_type is regex-guessed when the producer omits it (note: producer-declared type is honored first — plugin_metrics_publisher.ex:126-131). Two hardening items it left open and this work must close: the metrics stream still has no Nats-Msg-Id/duplicate_window, and two unrelated changes rode along (refresh_trace_summaries_worker chunk-size 3600→300, web-ng raw-traces fallback) that should be separated for clean revertability.

---

Recommendations (ordered, for the implementing agent)

1. Land additively, not as a v2 break. Reuse serviceradar.metric.v1, bump schema_version 1→2, add resource/kind/temporality/is_monotonic/points[]/exemplars as optional. Version-1 flat envelopes map to a single gauge point; the Telemetry processor keeps consuming both shapes.
2. Add the first-class metric emit path to both WASM SDKs first (signal_type:'metric' + payload_kind:'otel_metric' + NewMetricTelemetryRecord / TelemetryRecord::otel_metric via the existing emit_telemetry import). Keep AddMetric/add_metric_spec as perfdata shims so existing plugins keep working.
3. Make producers declare kind explicitly and gate the gateway regex inference to legacy version-1 envelopes only — eliminates the silent series-reshuffle and subject-token instability.
4. Add Nats-Msg-Id=ingress_id + a metrics-stream duplicate_window BEFORE enabling multi-point fan-out, and enforce idempotency in the TimescaleDB ingestors. At-least-once with consumer_max_deliver=5 will double-count otherwise.
5. Pick ONE point-identity recipe over (resource tuple + name + point.attributes) and converge both hypertable PKs (otel_metric_points recipe-v2 vs TimeseriesSeriesKey). This is a PK migration on both tables — sequence it deliberately.
6. Upgrade the anomaly consumer to be kind/temporality-aware (sample_extractor.ex/context_owner.ex): carry kind/temporality/unit through; rate-compute monotonic counters; ingest histogram count/sum/buckets for p50/p95 instead of rejecting value=nil. The contract is only worth its richness if a consumer uses it.
7. Migrate the gateway sysmon/snmp/icmp/mtr/sweep producers LAST, after the shared go/pkg/metricpoint builder lands: re-emit the legacy schemas as serviceradar.metric.v1 multi-point envelopes, collapsing the three legacy schemas into one and fixing the gateway drops at the same time.
8. Keep the addon raw-OTLP relay OTLP-native end to end (zero re-instrumentation for OTLP authors). Unify via a decode-and-republish shim. Decide two-lane (keep otel.metrics.raw as the high-fidelity express lane) vs one-lane.
9. Do NOT touch OCSF. Keep the documented separation; add the mis-bucket guardrails; continue emitting anomaly Findings (class 2004) and NetFlow Traffic (class 4001).
10. Add schema validation at gateway ingress and core consumer: reject absent/unknown schema instead of coercing to gauge; reject malformed serviceradar.metric.v1 rather than silently dropping non-numeric metrics.

---

Open questions

1. Wire encoding: keep serviceradar.metric.v1 as JSON (matches today, debuggable, loses int/double distinction, heavier) or move the high-rate/addon-volume path to protobuf closer to OTLP NumberDataPoint? Hybrid (JSON for plugins, protobuf for collectors) is plausible.
2. Two-lane vs one-lane for the addon raw-OTLP path: retire otel.metrics.raw in favor of the decode-and-republish shim, or keep it as the permanent high-fidelity express lane (exemplars/exp-histograms)?
3. Resource-identity trust: producer-claimed resource.attributes vs gateway-attested ingest_identity can disagree (a plugin reporting many guest devices under one mTLS agent). Which wins for series identity, and do we need a per-point attested device binding (the DIRE behavioral-identity concern) rather than trusting producer labels?
4. Single point-identity recipe: otel_metric_points recipe-v2 vs TimeseriesSeriesKey differ; which recipe, and what migration sequence for both PKs?
5. Consumer scope: does the anomaly/causal engine get upgraded to be kind/temporality-aware (rate counters, ingest histogram p50/p95) as part of this, or does the contract carry the fields while the consumer keeps collapsing to a scalar?
6. Histogram storage: otel_metric_points stores bucket_counts/explicit_bounds but drops exp-histograms/summaries; timeseries_metrics has no histogram columns. Extend timeseries_metrics or force all histograms to otel_metric_points?
7. Unit normalization ownership: producers emit UCUM and the gateway validates, or the gateway normalizes free-form units?
8. Sysmon per-family hypertables: retain cpu_metrics/memory_metrics/disk_metrics/process_metrics for query-shape reasons after sysmon emits serviceradar.metric.v1, or collapse into the unified sink? (Migrating onto the contract does not by itself mandate dropping the per-family tables.)
9. Metric descriptors: mapper/discovery emits per-interface metric definitions (InterfaceMetric), not samples. Does the contract carry metric-descriptors on a separate channel, or is the catalog out of scope?

---

Verification note

Findings were produced by parallel subsystem readers and an adversarial verification pass; load-bearing claims were re-checked against code. Three initial claims were REFUTED and corrected above: (a) metrics are NOT mis-bucketed into OCSF; (b) native add-ons DO have a friendly metric API (same plugin_result path as wasm) rather than only raw OTLP; (c) PluginMetricsPublisher honors a producer-declared metric_type first, regex is only the fallback. The producer sweep also corrected three sub-agent over-classifications (netprobe / flow-collector / otel-addon StreamTelemetry are operational self-metrics, not smuggled time-series).

## TL;DR A single CPU-utilization metric reaches the anomaly engine through **three structurally disjoint pipelines** (Go agent sysmon, WASM/native plugin via `plugin_result`, native add-on via OTLP relay) that converge **only** on shared gRPC transport (`AgentGatewayService.StreamStatus` with opaque JSON in `GatewayServiceStatus.Message`, disambiguated by an agent-set `Source` string) and shared `IngressId`/`Sr-Ingress-*` attribution headers — **never on schema, subject, point-identity, or storage**. Three series-identity algorithms and three DB sinks (`cpu_metrics`, `timeseries_metrics`, `otel_metric_points`) mean the same physical core produces three uncorrelatable anomaly series, and only the OTLP path is analyzed by default. The core defects, all verified against code: 1. **Metrics are smuggled inside status/check-result envelopes** because producers have no first-class metric emit path. The agent-gateway then does a **parse-forward-reparse-republish** dance — the same metric JSON is decoded up to **three times** (gateway publisher → core `StatusHandler` after `forward` → core `Metrics` processor off NATS). 2. **Metric-shape fragmentation** is the real disease, *not* OCSF: 4 wire schemas, 3 sinks, 3 non-alignable series-identity recipes, no shared point identity. 3. **No `kind` / `temporality` anywhere in the producer SDKs**, and the anomaly consumer **discards `kind`/`temporality`/`unit` even when the OTLP path supplies them** — a cumulative counter fed raw into z-sigma looks like a ramp. **OCSF is NOT mis-used.** ServiceRadar correctly keeps metric time-series out of OCSF classes; the separation is deliberate and documented (`elixir/serviceradar_core/lib/serviceradar/event_writer/ocsf.ex:16-20`). The fix is metric-shape unification, which is **orthogonal to OCSF** and must not be solved by routing into it. **The fix is additive, not a rewrite.** PR #3787 already shipped `serviceradar.metric.v1` as a thin bridge and is the de-facto reference implementation. Evolve it to an OTLP-grade contract (`schema_version 1→2`, add `resource`/`kind`/`temporality`/`points[]`), give all producers a first-class metric emit path, lower the legacy sysmon/snmp/shadow special-cases onto it, and delete the migration-era debt. --- ## Architectural principle > **Metrics are a first-class signal.** Producers (WASM plugins, native add-ons, and the agent sysmon/snmp/icmp/mtr/sweep/rperf collectors) emit one `serviceradar.metric.v1` shape **directly onto a metric path** — never embedded inside a status / check-result / plugin-result envelope. The gateway stops re-extracting metrics from statuses, and the per-source (`sysmon`/`snmp`/`plugin`) special-cases collapse into one emit + one ingest + one sink. --- ## Current state — four disjoint pipelines (CPU metric traced end-to-end) | | Go agent sysmon | WASM/native plugin | native add-on OTLP relay | |---|---|---|---| | **producer shape** | `MetricSample.cpus[]{core_id,usage_percent}` (`go/pkg/sysmon/metrics.go:25-180`) | `Metric{name,value,unit,warn,crit,min,max}` (`serviceradar-sdk-go/sdk/result.go:52-60`; Rust `src/result.rs:138-184`) | OTLP `NumberDataPoint` in `ExportMetricsServiceRequest` | | **wrap / source** | JSON in `GatewayServiceStatus.Message`, `Source="sysmon-metrics"` (`push_loop_status.go:200-233`) | embedded in `serviceradar.plugin_result.v1` over `env.submit_result`, `Source="plugin-result"` | opaque OTLP, `Source="otlp-relay"` | | **gateway publisher** | `SysmonMetricsPublisher` → `serviceradar.sysmon.metrics.v1` → `metrics.sysmon.<family>` | `PluginMetricsPublisher` → `serviceradar.metric.v1` → `metrics.timeseries.<type>.<name>` | `OtlpRelayPublisher` → verbatim → `otel.metrics.raw` | | **stream** | `metrics` | `metrics` | `events` (separate) | | **core processor → sink** | `Metrics`→`SysmonMetricsIngestor` → **`cpu_metrics`** per-family hypertable | `Metrics`→`Telemetry` → **`timeseries_metrics`** | `OtelMetrics` (protobuf) → **`otel_metric_points`** | | **anomaly series id** | `sysmon:cpu:<host>:<core_id>` | `cpu:<TimeseriesSeriesKey md5>` | `otel:<svc>:<name>:<attributes_hash>` | | **kind/temporality/unit** | none | none (unit stored, then dropped) | **full — then discarded by consumer** | The gateway is **not a normalizer** — it is four source-keyed pass-through publishers selected by the agent-supplied `Source` string (`status_processor.ex:57-97,328-431`). ### Divergences (why the same core yields three uncorrelatable series) - **3 series-identity algorithms**, none alignable: sysmon string-built (`sample_extractor.ex` ~`:123`), plugin `TimeseriesSeriesKey` md5 (`timeseries_series_key.ex:6-35`), OTLP `attributes_hash` recipe-v2 (`otel_metric_point.ex:19-198`). - **3 DB sinks / schemas** with no unifying view or join (repo-wide search found no UNION/view merging `timeseries_metrics` and `otel_metric_points`). - **3 wire schemas / encodings**: `serviceradar.sysmon.metrics.v1` (whole-sample blob), `serviceradar.metric.v1` (per-scalar JSON), raw OTLP protobuf. - **Coverage asymmetry**: anomaly `enabled_subjects` defaults to `['otel.metrics.>']` only (`anomaly_detection/config.ex:18`), so **only the native-addon OTLP CPU metric is analyzed out of the box**; sysmon and plugin CPU are subscribed but gated off until `ANOMALY_ANALYSIS_ENABLED_SUBJECTS` is widened. - **Per-device identity collapse** on the plugin path only: `serviceradar.metric.v1` never sets `device_id`/`target_device_ip`/`if_index` and per-metric labels are not promoted to tags, so many devices under one agent collide into one series (`telemetry.ex:112-137`). - **Timestamp semantics differ**: OTLP has true per-point `time_unix_nano`; sysmon shares one sample-level timestamp across all cores; plugin shares the `Result` envelope `observed_at` across all metrics. - **Consumer destroys fidelity uniformly**: `SampleExtractor → ContextOwner → CausalReasoner` reduces every path to `{value, observed_at_unix_nano}` (`sample_extractor.ex`, `context_owner.ex:349-363`). `kind`/`temporality`/`unit`/`is_monotonic` never reach the reasoner even when `otel_metric_points` has them. --- ## The double/triple-decode (the "re-publish" smell) `status_processor.ex` `forward_then_publish/1` (`:78-97`) does two passes over the same payload: (1) `forward(status)` ships the whole status to core for status/state processing, then (2) re-parses `status.message` and republishes the extracted metrics onto JetStream via three source-specific publishers. Evidence of the same metric JSON decoded three times: - `sysmon_metrics_publisher.ex:42-51` / `snmp_metrics_publisher.ex:44-53` / `plugin_metrics_publisher.ex:44-53` — each `Jason.decode(status.message)` (decode #2). - `metrics.ex:46-72` — core re-decodes the republished NATS message (decode #3). It is **not** a duplicate DB write today: the direct-insert sink is flag-gated off (`plugin_result_ingestor.ex:6-10,:52,:200-206`, `:plugin_result_direct_metrics_enabled` default `false`), and the core gRPC-direct-to-DB metric handlers are already no-op stubs (`results_router.ex:239-245,:611` → `acknowledge_metrics_cutover/1 → :ok`). So half the migration is already done — it just hasn't been cleaned up. The remaining smell is purely the smuggle-and-re-extract architecture, which the first-class metric path eliminates. --- ## OCSF finding — NOT a category error (keep the separation, add guardrails) This was investigated directly and the "metrics are mis-bucketed into OCSF" claim was **REFUTED** across producers, transport, consumers, and schema: - Policy is documented in code: `ocsf.ex:16-20` — *"OTel traces/metrics: Keep native format (observability, not security)"* and *"Telemetry metrics: Keep native format (time-series data)"*. - `ocsf_events` (migration `20260203120000_create_ocsf_events.exs:14-45`) is a pure security-event shape (`class_uid`/`category_uid`/`activity_id`/`severity_id`/observables/actor/device/endpoints) with **zero** metric value/unit/temporality/metric_name columns — it physically cannot hold a series. - Both WASM SDKs only have `ocsf_event`/`otel_log` payload kinds (`serviceradar-sdk-go/sdk/signal_schema.go:17-18`, `serviceradar-sdk-rust/src/result.rs:356-357`) — there is no metric signal type, so a metric cannot be serialized as an OCSF event. - The only numbers legitimately inside OCSF are (a) NetFlow byte/packet counters as `traffic{}` in `ocsf_network_activity` class 4001 (event-shaped, OCSF-sanctioned), and (b) anomaly/capacity **Findings** (class 2004) carrying a single anomalous value + thresholds as event context (`verdict_emitter.ex:34-74,:96-118`) — derived *from* metrics, not duplicating the series. Both are correct and stay. **Recommendations:** - **Do NOT add a "metric" OCSF class.** OCSF has no metric/time-series primitive; inventing one would be the actual category error. - **Keep using OCSF for events derived from metrics** (threshold breaches → Findings) — the code already does this correctly. - **Add mis-bucket guardrails** with the unified contract: the metrics processor rejects any `serviceradar.metric.v1` payload carrying `class_uid`; the OCSF processor rejects payloads carrying `kind`/`temporality`/`points[]`. A misrouted message then errors loudly instead of landing in the wrong hypertable. --- ## Confirmed gaps (verified against code) 1. **No metric `kind`** (gauge/counter/histogram) in any producer SDK or in three of four gateway envelopes — `result.go:52-60`, `result.rs:138-184`. Downstream `metric_type` is a routing label, not an OTel kind. 2. **No `temporality`** (delta/cumulative) in SDKs or the `serviceradar.metric.v1` / `serviceradar.sysmon.metrics.v1` envelopes (`plugin_metrics_publisher.ex:93-111`). SNMP carries only `is_delta`. Result: monotonic counters look like ramps to z-sigma. 3. **No first-class metric emit helper in either WASM SDK** — `telemetry.go` has only `NewOCSFTelemetryRecord`(`:36`) and `NewOTELLogTelemetryRecord`(`:53`); `telemetry.rs` only `ocsf_event`(`:60`)/`otel_log`(`:76`). Metrics can only ride embedded in `submit_result`; there is no metric payload kind. 4. **No per-metric tags / resource identity / point identity** on the `plugin_result` `Metric` shape — labels are envelope-level (`result.go:37`, `result.rs:385`); per-metric dimensions are lost (`plugin_metrics_publisher.ex:113-191`). 5. **Per-device identity collapse on the plugin path** — `serviceradar.metric.v1` never sets `device_id`/`target_device_ip`/`if_index`, so many devices under one agent collapse into one series. 6. **`metrics.timeseries.*` and `otel.metrics.raw` decode into different sinks** (`timeseries_metrics` vs `otel_metric_points`) with no canonical merge. 7. **Anomaly consumer drops `kind`/`temporality`/`unit`** at extraction even when OTLP supplies them (`sample_extractor.ex`, `context_owner.ex:349-363`). 8. **Coverage losses inside the (correct) metric namespace**: OTLP exponential histograms & summaries are counted-then-dropped (`otel_metrics.ex` ~`:258-281`); the SNMP gateway publisher drops every OID except `ifHCInOctets`/`ifHCOutOctets` (`snmp_metrics_publisher.ex`); sysmon `Network[]` interface metrics are collected but dropped at the gateway (only cpu/memory/disk/process families emitted). 9. **No `Nats-Msg-Id` / `duplicate_window` on the `metrics` stream** (`config.ex:206-219`). With `consumer_max_deliver=5` and the planned multi-point fan-out, at-least-once delivery would double-count; idempotency must be enforced at the stream and/or ingestor. 10. **No schema/proto validation of `serviceradar.metric.v1`** — the producer hand-builds the map and the consumer tolerantly defaults missing fields (`metric_type→'gauge'`); malformed envelopes become `gauge`/`unknown` rows instead of being rejected. Non-numeric metrics are silently dropped (`encode_metric` returns `{:ok, nil}` with no counter — invisible data loss). --- ## Producer refactor work-list (every producer smuggling metrics through status) Swept all 12 native add-ons, both WASM SDKs, and in-tree first-party checkers (22 producers classified). ### `SMUGGLES_METRICS` — first-party in-tree checkers (the real refactor targets) | producer | metrics | source string | evidence | |---|---|---|---| | **sysmon** | cpu/disk/memory/network-iface/process | `sysmon-metrics` | `go/pkg/sysmon/metrics.go:25-180`, `push_loop_status.go:200-233` | | **snmp** | OID counters/gauges w/ `if_index`,`interface_uid` | `snmp-metrics` | `push_loop_snmp.go:30-42,48-57,63` | | **icmp** | `response_time_ns`, `packet_loss` | `results` | `push_loop_icmp_results.go:29-68`, `icmp_checker.go:69-77` | | **mtr** | per-hop RTT/loss/jitter | `results` | `mtr_checker.go:61-70,86-127`, `proto/monitoring.proto:1012-1046` | | **network sweep** | scanner perf counters (`SweepScannerStats`) | `results` | `proto/monitoring.proto:149-200`, `push_loop_sweep_results.go:28-118` | | **mapper/discovery** | per-interface metric **descriptors** (`InterfaceMetric`) — catalog, not samples | discovery status | `proto/discovery/discovery.proto:188-214`, `push_loop_mapper_netprobe.go:36-93` | | **rperf** (Rust checker) | throughput/loss/jitter (`TestSummary`) | `plugin-result` (generic path) | `proto/rperf/rperf.proto:60-72`, `rust/rperf-client/src/server.rs:259-300` | > Note: mapper/discovery carries metric **definitions** (the catalog the snmp checker samples), not time-series — decide whether the unified contract carries metric-descriptors on a separate channel. ### SDK smuggling helpers (add first-class emit here, then deprecate) - Go: `Result.Metrics` slot (`serviceradar-sdk-go/sdk/result.go:36`), `Metric` struct (`:52-60`), `AddMetric`/`WithMetric` builders (`:257-273`). - Rust: `Metric` struct (`serviceradar-sdk-rust/src/result.rs:138-151`), `add_metric_spec`/`with_metric_spec` (`:578-585`), `add_metric`/`with_metric` (`:587-612`). - 6 SDK examples (`http-check`, `udp-check`, `tcp-check`, `widgets-check` ×Go; `http-check`, `widgets-check` ×Rust) and 2 golden fixtures (`testdata/service_monitoring_result.json` ×2) demonstrate the pattern and must update in lockstep. ### `OTLP_NATIVE` (already first-class) - **otel-addon** + `rust/otel` library relay customer OTLP metrics via the acked `RelayOtlp` stream (`otlp-relay:v1`): `rust/otel-addon/src/addon.rs:27,556`, `rust/otel/src/lib.rs:529,754`, `addon.proto:159-165` (`OTLP_METRICS=5`, `OTLP_DERIVED_METRIC=6`). Decision needed: normalize OTLP_METRICS frames into `serviceradar.metric.v1` downstream, or keep `otel.metrics.raw` as a high-fidelity express lane. ### `NO_METRICS` (no refactor) — explicitly verified to avoid false positives bumblebee-scan, endpoint-inventory, scalibr-endpoint-inventory, advisory-producer, sample-addon, powerdns, rdp-adapter, workload-identity, bmp-collector, log-collector, trapd, flowgger, rust-sample-addon, rperf-server. **Correction to parallel sub-agents:** netprobe, flow-collector, and otel-addon's `StreamTelemetry` expose only **operational Prometheus/OCSF-usage self-metrics**, NOT smuggled observability time-series — they are NOT refactor targets. --- ## Tech-debt to delete / collapse (the unification makes the special-cases deletable) ### DELETE (pure migration-era scaffolding) - `@legacy_sysmon_schema "serviceradar.sysmon.shadow.v1"` acceptance — `metrics.ex:19,:151`. - `AGENT_GATEWAY_SYSMON/SNMP_METRICS_SHADOW_ENABLED` env defaults (`runtime.exs:314,318`, default `"true"`) + helm `*MetricsShadowEnabled` aliases (`agent-gateway.yaml:30,36`). - Flag-gated legacy direct insert `maybe_insert_metrics` + `:plugin_result_direct_metrics_enabled` (`plugin_result_ingestor.ex:52,:200-206`; `core/config.exs:131-133`). Keep `build_status_row`/`insert_status` — only the metric insert is debt. - Vestigial no-op router handlers `handle_sysmon_metrics`/`handle_snmp_metrics`/`acknowledge_metrics_cutover` (`results_router.ex:239-245,:611`) and their dispatch entries (`status_handler.ex:109-111`). ### COLLAPSE (replace source-specific handling with the unified path) - `SysmonMetricsPublisher` (`sysmon_metrics_publisher.ex` whole module) and `SnmpMetricsPublisher` (`snmp_metrics_publisher.ex` whole module) → fold into a generalization of `PluginMetricsPublisher` (which already emits `serviceradar.metric.v1`). These are the publish boundary (agents are NATS-denied), so they must be **replaced, not merely deleted**. - Sysmon family-envelope carve-out in `metrics.ex:18,50,59-181` (`parse_sysmon`/`filter_sysmon_sample`/`partition_messages`/`ingest_sysmon_messages`/`metric_kind`) → the generic Telemetry/timeseries path. - `SysmonMetricsIngestor` (`sysmon_metrics_ingestor.ex`) 5-hypertable fan-out → unified ingest. - Gateway double-pass `forward_then_publish` + `publish_sysmon/snmp/plugin_metrics` + `*_source?` guards (`status_processor.ex:78-97,313-427`); per-source `max_message_bytes`/`strict_message_size_source?` (`agent_gateway_server.ex:665-680`) — **preserve the large sysmon size budget** on the unified path. - Anomaly per-source carve-outs: `sample_extractor.ex:67-71,114-189`, `series_config.ex:201-214`, `config.ex:108-109` (`ANALYSIS_METRICS_SYSMON`/`SNMP`). - Per-source subject prefixes / NATS ACLs (`config.exs:30,35`, `runtime.exs:346,351`, `agent-gateway.yaml:127,131`, `nats.yaml:211`, `values-demo.yaml:488`, `values.yaml:946,948`) — re-bind stream subjects + durable consumer filters **atomically** during cutover. ### MIGRATE (producer code switches to first-class emit) - Go agent sysmon producer (`push_loop_status.go:118-233`, `sysmon_service.go:604-612`, `go/pkg/sysmon/{metrics,collector}.go`) — flatten cpus/clusters/disks/processes into per-series scalar points. - Go agent SNMP producer (`push_loop_snmp.go:66,77,101,140`) — keep `if_index`/`target_device_ip` identity. - (+ icmp/mtr/sweep/rperf per the producer work-list above.) ### KEEP — do NOT delete (sysmon-named but config/UI, not metric debt) - `sysmon_profiles` table + migrations (`20260519162000_*`, `20260123093000_*`), `sysmon_compiler.ex`, settings UI (`sysmon_profiles_live/index.ex`) — **configuration**. - Device-detail sysmon charts UI (`device_tab_runtime.ex`) and SRQL catalog — **repoint** to the unified table, do not delete. - `go/pkg/sysmon/collector.go` + `sysmon_service.go` — the collector itself (migrate the emit shape, not the collector). - The five sysmon **metric** hypertables (`cpu_metrics`/`cpu_cluster_metrics`/`memory_metrics`/`disk_metrics`/`process_metrics`) + `_hourly` caggs — unification candidates, but they back live UI charts and anomaly baselines: **data-migrate before any DROP**. ### Tests that lock in the per-source schemas/subjects (will need rewriting) `metrics_test.exs` (legacy shadow + sysmon/snmp fixtures), `sysmon_metrics_publisher_test.exs`, `snmp_metrics_publisher_test.exs`, `status_processor_test.exs:428,456`, `sysmon_metrics_ingestor_test.exs`, `snmp_metrics_ingestor_integration_test.exs`, `results_router_test.exs:431,445`, `sample_extractor_test.exs`, `pipeline_test.exs`, `config_test.exs:40-88`, `series_config_test.exs`/`synthetic_dataset_test.exs`/`verdict_emitter_test.exs`/`baseline_seeder_test.exs`/`stateful_alert_engine_test.exs`, `jetstream_consumer_test.exs`, `pipeline_ack_test.exs`/`pipeline_batch_span_test.exs`. --- ## Proposed unified contract: `serviceradar.metric.v1` (evolve additively from the PR #3787 bridge) Reuse the existing schema id; bump `schema_version` 1→2 and add the OTLP-grade fields below. Version-1 flat envelopes remain valid (treated as a single gauge point: `value→points[0].value`, `metric_type→kind`, `tags→points[0].attributes`, `observed_at→points[0].time_unix_nano`). Consumers MUST reject envelopes whose `schema` is absent/unknown rather than coercing to `gauge`. | field | type | purpose | OTLP equiv | |---|---|---|---| | `schema` | const `"serviceradar.metric.v1"` | route + version-gate | Schema URL analog | | `schema_version` | uint32 (default 1) | additive evolution | — | | `resource` | `{service_name(req), service_instance_id, scope_name, scope_version, attributes:map}` | resource identity carried once per batch/point — the field the SDK `Metric` and PR3787 envelope both lack. sysmon: `service_name='sysmon'`, attrs carry `host_id`/`core_id`/`mount_point`/`pid`; snmp: `target_device_ip`/`if_index`/`interface_uid` | Resource + InstrumentationScope | | `name` | string (req) | metric stream name (dotted semconv), no longer the sole identity | Metric.name | | `kind` | enum `{gauge, sum, histogram, exp_histogram, summary}` | the field every producer is missing; lets the engine rate-compute counters | Metric.data oneof | | `temporality` | enum `{unspecified, delta, cumulative}` (req for sum/histogram) | counter-rate computability; snmp `is_delta→delta` | aggregation_temporality | | `is_monotonic` | bool (kind=sum) | counter vs UpDownCounter | Sum.is_monotonic | | `unit` | string (UCUM: `%`,`By`,`1`,`ms`,`Hz`) | stops n_sigma mixing percent/bytes/counts | Metric.unit | | `points` | array<DataPoint> (req, ≥1) | one OR MANY points per emit so cores/mounts/interfaces/guests dimension **without** splitting envelopes (fixes per-device collapse) | data_points (repeated) | | `points[].time_unix_nano` | uint64 (req) | per-point measurement time | DataPoint.time_unix_nano | | `points[].start_time_unix_nano` | uint64 (req for cumulative rate) | counter reset/window anchor | DataPoint.start_time_unix_nano | | `points[].attributes` | map<string,scalar> | per-point dims (`core_id`,`mount_point`,`if_index`,`guest_id`); volatile keys excluded from identity | DataPoint.attributes | | `points[].value` | double\|int (gauge/sum) | int vs double preserved | NumberDataPoint.as_double/as_int | | `points[].histogram` | `{count,sum,bucket_counts[],explicit_bounds[],min,max}` (histogram) | first-class histogram (none today); enables p50/p95 series currently dropped | HistogramDataPoint | | `points[].exemplars` | `[{time_unix_nano,value,trace_id,span_id,filtered_attributes}]` (opt) | metric→trace pivots the causal engine wants | DataPoint.exemplars | | `thresholds` | `{warn,crit,min,max}` (opt) | preserves Nagios perfdata semantics **without** conflating with metric data; alerting/display only | — (deliberately outside OTLP) | | `ingress_id` | UUIDv8 (gateway-stamped) | per-message ordering/idempotency; also set as `Nats-Msg-Id` | — | | `ingress_timestamp_unix_nano` | uint64 (gateway-stamped) | ingest/transport time, distinct from measurement time | — | | `ingest_identity` | `{ingest_agent_id, ingest_partition, ingest_identity:'agent:<id>'}` (gateway-attested) | mTLS-bound attested identity, separate from producer-claimed `resource.attributes` (untrusted) | attested overlay | ### Per-SDK emit APIs (new third signal stream: `signal_type:'metric'` / `payload_kind:'otel_metric'`) - **go-wasm**: `sdk.NewGauge/NewCounter/NewHistogram(name, value, ...MetricOption)` with `WithUnit/WithAttributes/WithTimestamp/WithResource/WithThresholds/WithExemplar`; collected into `sdk.NewMetricBatch(resource)`; add `NewMetricTelemetryRecord(batch)` so metrics ride `env.emit_telemetry`. Keep `Result.AddMetric` as a shim lowering to a gauge point. - **rust-wasm**: `MetricPoint::gauge/counter/histogram(...)` + `MetricBatch::new(ResourceRef::service(...))` + `TelemetryRecord::otel_metric(batch)`. Keep `Metric`/`add_metric_spec` as a perfdata shim. - **native add-on** (`rust/addon-sdk`): `MetricBatchBuilder::new(resource).gauge/counter/histogram(...)` + `Addon::emit_metrics(batch)` wrapping a new `TelemetryPayloadKind::SR_METRIC_V1` (alongside `OTLP_METRICS=5`, `OTLP_DERIVED_METRIC=6`). OTLP-native authors keep `relay_otlp()` unchanged. - **agent sysmon/snmp/icmp/mtr/sweep**: internal `go/pkg/metricpoint` builder mirroring the SDK so the gateway publishers emit `serviceradar.metric.v1` multi-point envelopes (one `Metric{kind,unit,points[]}` per family), replacing the raw-sample blob and the 2-OID hardcode; emit the currently-dropped sysmon `Network[]` and SNMP non-octet OIDs. - **shared**: `ResourceRef.service(name).instance(id).scope(name,ver).attr(k,v)` so all producers construct the resource tuple once. ### Subject topology One canonical tree `metrics.v1.<kind>.<domain>.<name-token>` on the existing high-rate `metrics` stream (widen the binding to also own `metrics.v1.>`). Leading token = the real `kind` field (no longer regex-guessed). Legacy subjects (`metrics.sysmon.*`, `metrics.snmp.interface.*`, `metrics.timeseries.<type>.<name>`) stay bound during migration. The addon raw-OTLP path is NOT moved by alias — a decode-and-republish shim lowers `ExportMetricsServiceRequest` into `metrics.v1.*` for a single consumer shape (or keep `otel.metrics.raw` as the express lane — see open questions). **Critical hardening before fan-out:** gateway sets `Nats-Msg-Id`=`ingress_id` on every publish AND the `metrics` stream gains a `duplicate_window`. Add `metrics.v1.>` to `ANOMALY_ANALYSIS_ENABLED_SUBJECTS`. --- ## Relationship to PR #3787 PR #3787 is a correct, deliberately-thin **bridge** that should be **evolved, not replaced**. It ships `serviceradar.metric.v1` as a flat `{schema, metric_name, metric_type, value, unit, tags, metadata}` envelope on `metrics.timeseries.<type>.<name>` → Telemetry → `timeseries_metrics`, wires the anomaly consumer (`ANALYSIS_METRICS_TIMESERIES`), and cleanly gates off the legacy direct-DB insert. It is the de-facto reference implementation of the unified contract. It explicitly does **not** solve the resource-metric SDK contract: no kind/temporality, no resource identity (`device_id`/`if_index`/`target_device_ip` always nil → per-device collapse), no per-metric labels, and `metric_type` is regex-guessed when the producer omits it (note: producer-declared type **is** honored first — `plugin_metrics_publisher.ex:126-131`). Two hardening items it left open and this work must close: the `metrics` stream still has no `Nats-Msg-Id`/`duplicate_window`, and two unrelated changes rode along (`refresh_trace_summaries_worker` chunk-size 3600→300, web-ng raw-traces fallback) that should be separated for clean revertability. --- ## Recommendations (ordered, for the implementing agent) 1. **Land additively, not as a v2 break.** Reuse `serviceradar.metric.v1`, bump `schema_version 1→2`, add `resource`/`kind`/`temporality`/`is_monotonic`/`points[]`/`exemplars` as optional. Version-1 flat envelopes map to a single gauge point; the Telemetry processor keeps consuming both shapes. 2. **Add the first-class metric emit path to both WASM SDKs first** (`signal_type:'metric'` + `payload_kind:'otel_metric'` + `NewMetricTelemetryRecord` / `TelemetryRecord::otel_metric` via the existing `emit_telemetry` import). Keep `AddMetric`/`add_metric_spec` as perfdata shims so existing plugins keep working. 3. **Make producers declare `kind` explicitly** and gate the gateway regex inference to legacy version-1 envelopes only — eliminates the silent series-reshuffle and subject-token instability. 4. **Add `Nats-Msg-Id`=`ingress_id` + a `metrics`-stream `duplicate_window` BEFORE enabling multi-point fan-out**, and enforce idempotency in the TimescaleDB ingestors. At-least-once with `consumer_max_deliver=5` will double-count otherwise. 5. **Pick ONE point-identity recipe** over `(resource tuple + name + point.attributes)` and converge both hypertable PKs (`otel_metric_points` recipe-v2 vs `TimeseriesSeriesKey`). This is a PK migration on both tables — sequence it deliberately. 6. **Upgrade the anomaly consumer to be kind/temporality-aware** (`sample_extractor.ex`/`context_owner.ex`): carry `kind`/`temporality`/`unit` through; rate-compute monotonic counters; ingest histogram count/sum/buckets for p50/p95 instead of rejecting `value=nil`. The contract is only worth its richness if a consumer uses it. 7. **Migrate the gateway sysmon/snmp/icmp/mtr/sweep producers LAST**, after the shared `go/pkg/metricpoint` builder lands: re-emit the legacy schemas as `serviceradar.metric.v1` multi-point envelopes, collapsing the three legacy schemas into one and fixing the gateway drops at the same time. 8. **Keep the addon raw-OTLP relay OTLP-native end to end** (zero re-instrumentation for OTLP authors). Unify via a decode-and-republish shim. Decide two-lane (keep `otel.metrics.raw` as the high-fidelity express lane) vs one-lane. 9. **Do NOT touch OCSF.** Keep the documented separation; add the mis-bucket guardrails; continue emitting anomaly Findings (class 2004) and NetFlow Traffic (class 4001). 10. **Add schema validation** at gateway ingress and core consumer: reject absent/unknown `schema` instead of coercing to `gauge`; reject malformed `serviceradar.metric.v1` rather than silently dropping non-numeric metrics. --- ## Open questions 1. **Wire encoding**: keep `serviceradar.metric.v1` as JSON (matches today, debuggable, loses int/double distinction, heavier) or move the high-rate/addon-volume path to protobuf closer to OTLP `NumberDataPoint`? Hybrid (JSON for plugins, protobuf for collectors) is plausible. 2. **Two-lane vs one-lane** for the addon raw-OTLP path: retire `otel.metrics.raw` in favor of the decode-and-republish shim, or keep it as the permanent high-fidelity express lane (exemplars/exp-histograms)? 3. **Resource-identity trust**: producer-claimed `resource.attributes` vs gateway-attested `ingest_identity` can disagree (a plugin reporting many guest devices under one mTLS agent). Which wins for series identity, and do we need a per-point attested device binding (the DIRE behavioral-identity concern) rather than trusting producer labels? 4. **Single point-identity recipe**: `otel_metric_points` recipe-v2 vs `TimeseriesSeriesKey` differ; which recipe, and what migration sequence for both PKs? 5. **Consumer scope**: does the anomaly/causal engine get upgraded to be kind/temporality-aware (rate counters, ingest histogram p50/p95) as part of this, or does the contract carry the fields while the consumer keeps collapsing to a scalar? 6. **Histogram storage**: `otel_metric_points` stores bucket_counts/explicit_bounds but drops exp-histograms/summaries; `timeseries_metrics` has no histogram columns. Extend `timeseries_metrics` or force all histograms to `otel_metric_points`? 7. **Unit normalization ownership**: producers emit UCUM and the gateway validates, or the gateway normalizes free-form units? 8. **Sysmon per-family hypertables**: retain `cpu_metrics`/`memory_metrics`/`disk_metrics`/`process_metrics` for query-shape reasons after sysmon emits `serviceradar.metric.v1`, or collapse into the unified sink? (Migrating onto the contract does not by itself mandate dropping the per-family tables.) 9. **Metric descriptors**: mapper/discovery emits per-interface metric *definitions* (`InterfaceMetric`), not samples. Does the contract carry metric-descriptors on a separate channel, or is the catalog out of scope? --- ## Verification note Findings were produced by parallel subsystem readers and an adversarial verification pass; load-bearing claims were re-checked against code. Three initial claims were **REFUTED** and corrected above: (a) metrics are NOT mis-bucketed into OCSF; (b) native add-ons DO have a friendly metric API (same `plugin_result` path as wasm) rather than only raw OTLP; (c) `PluginMetricsPublisher` honors a producer-declared `metric_type` first, regex is only the fallback. The producer sweep also corrected three sub-agent over-classifications (netprobe / flow-collector / otel-addon `StreamTelemetry` are operational self-metrics, not smuggled time-series).

mfreeman451 referenced this issue 2026-06-13 20:21:45 +00:00

Host/system (sysmon) metric collection needs first-class monotonic-counter support (wrap + reset/reboot detection); SNMP is the partial-but-flawed reference #3789

mfreeman451 commented 2026-06-13 20:23:11 +00:00

Author

Owner

Copy link

Companion issue filed: #3789 — Host/system (sysmon) metric collection needs first-class monotonic-counter support (wrap + reset/reboot detection); SNMP is the partial-but-flawed reference.

It is the collector-level + algorithm deep-dive for the kind / is_monotonic / temporality / points[].start_time_unix_nano fields proposed here: how the agent's two metric collectors (go/pkg/sysmon/ and go/pkg/agent/snmp/) must actually produce correct counter semantics, why both get it wrong today (sysmon ships raw since-boot counters with no rate → perpetual z-score ramp; SNMP turns every reboot into a ~4.29B false spike and discards the raw value), and the reset-vs-wrap detection algorithm to adopt.

Companion issue filed: #3789 — *Host/system (sysmon) metric collection needs first-class monotonic-counter support (wrap + reset/reboot detection); SNMP is the partial-but-flawed reference.* It is the collector-level + algorithm deep-dive for the `kind` / `is_monotonic` / `temporality` / `points[].start_time_unix_nano` fields proposed here: how the agent's two metric collectors (`go/pkg/sysmon/` and `go/pkg/agent/snmp/`) must actually *produce* correct counter semantics, why both get it wrong today (sysmon ships raw since-boot counters with no rate → perpetual z-score ramp; SNMP turns every reboot into a ~4.29B false spike and discards the raw value), and the reset-vs-wrap detection algorithm to adopt.

mfreeman451 referenced this issue 2026-06-13 20:41:06 +00:00

Capability: cgroup-v2 per-tenant container resource metrics (per-account io/cpu/memory accounting) #3790

mfreeman451 referenced this issue 2026-06-13 20:47:24 +00:00

spec: define anomaly scale architecture #3791

mfreeman451 referenced this issue 2026-06-13 21:07:31 +00:00

work/v1.3.2-observability-metrics #3787

mfreeman451 referenced this issue 2026-06-13 21:07:31 +00:00

spec: define anomaly scale architecture #3791

mfreeman451 referenced this issue 2026-06-13 21:07:32 +00:00

spec: add counter and cgroup metric proposals #3793

mfreeman451 commented 2026-06-13 23:04:45 +00:00

Author

Owner

Copy link

Merged related PRs into staging:

• #3787 plugin metrics / observability fixes
• #3791 anomaly scale architecture OpenSpec
• #3792 compact evaluator benchmark/test scaffold
• #3793 counter + cgroup OpenSpec proposals
• #3794 cumulative counter normalizer
• #3795 SNMP raw counter semantics

Keeping this issue open because the full unified metric ingestion contract is not complete yet: schema v2, producer migrations, canonical SDK emit APIs, stream idempotency, and cleanup still need implementation.

Merged related PRs into staging: - #3787 plugin metrics / observability fixes - #3791 anomaly scale architecture OpenSpec - #3792 compact evaluator benchmark/test scaffold - #3793 counter + cgroup OpenSpec proposals - #3794 cumulative counter normalizer - #3795 SNMP raw counter semantics Keeping this issue open because the full unified metric ingestion contract is not complete yet: schema v2, producer migrations, canonical SDK emit APIs, stream idempotency, and cleanup still need implementation.

mfreeman451 referenced this issue 2026-06-13 23:30:17 +00:00

Anomaly ingest: derive p50/p95 from OTLP histogram points (stop dropping value=nil) [#3788 REC6b] #3800

mfreeman451 referenced this issue 2026-06-13 23:30:17 +00:00

Remove legacy sysmon shadow/cutover metric tech-debt (post-JetStream migration) [#3788] #3801

mfreeman451 referenced this issue from carverauto/serviceradar-sdk-rust 2026-06-13 23:30:25 +00:00

Add a first-class metric emit path to the WASM plugin SDK (signal_type 'metric' / payload_kind 'otel_metric') [serviceradar#3788 REC2] #5

mfreeman451 referenced this issue from a commit 2026-06-13 23:38:50 +00:00

fix(agent-gateway): surface dropped plugin metrics via telemetry

mfreeman451 referenced this issue 2026-06-13 23:39:10 +00:00

fix(agent-gateway): surface dropped plugin metrics via telemetry [#3788 REC10] #3802

mfreeman451 referenced this issue from a commit 2026-06-14 00:05:21 +00:00

feat(event-writer): reject mis-bucketed metric/OCSF payloads [#3788 REC9]

mfreeman451 referenced this issue 2026-06-14 00:05:41 +00:00

feat(event-writer): reject mis-bucketed metric/OCSF payloads [#3788 REC9] #3803

mfreeman451 referenced this issue 2026-06-14 00:16:31 +00:00

feat(metrics): dedup the metrics JetStream stream via Nats-Msg-Id [#3788 REC4] #3804

mfreeman451 referenced this issue from a commit 2026-06-14 00:16:31 +00:00

feat(metrics): dedup the metrics JetStream stream via Nats-Msg-Id [#3788 REC4]

mfreeman451 referenced this issue from a commit 2026-06-14 00:20:23 +00:00

feat(metrics): carry OTLP-grade v2 semantics in plugin metric envelope [#3788 REC1]

mfreeman451 referenced this issue 2026-06-14 00:20:24 +00:00

feat(metrics): carry OTLP-grade v2 semantics in plugin metric envelope [#3788 REC1] #3805

mfreeman451 referenced this issue from a commit 2026-06-14 00:25:03 +00:00

feat(event-writer): surface silent metric_type->gauge coercion [#3788 REC10]

mfreeman451 referenced this issue 2026-06-14 00:25:04 +00:00

feat(event-writer): surface silent metric_type->gauge coercion [#3788 REC10] #3806

mfreeman451 referenced this issue from a commit 2026-06-14 00:30:01 +00:00

feat(snmp): make the gateway interface-metric allowlist configurable [#3788 REC7e]

mfreeman451 referenced this issue 2026-06-14 00:30:02 +00:00

feat(snmp): make the gateway interface-metric allowlist configurable [#3788 REC7e] #3807

mfreeman451 referenced this issue from a commit 2026-06-14 00:32:55 +00:00

feat(metrics): record whether plugin metric_type was inferred [#3788 REC3]

mfreeman451 referenced this issue 2026-06-14 00:32:56 +00:00

feat(metrics): record whether plugin metric_type was inferred [#3788 REC3] #3808

mfreeman451 referenced this issue from a commit 2026-06-14 00:37:03 +00:00

feat(metricpoint): add shared serviceradar.metric.v1 builder for producers [#3788 REC7]

mfreeman451 referenced this issue 2026-06-14 00:37:39 +00:00

feat(metricpoint): shared serviceradar.metric.v1 builder for producers [#3788 REC7] #3809

mfreeman451 referenced this issue from a commit 2026-06-14 00:39:30 +00:00

chore(metrics): remove dead serviceradar.sysmon.shadow.v1 acceptance [#3788]

mfreeman451 referenced this issue 2026-06-14 00:39:32 +00:00

chore(metrics): remove dead serviceradar.sysmon.shadow.v1 acceptance [#3788] #3810

mfreeman451 referenced this issue 2026-06-14 00:48:42 +00:00

chore(observability): remove flag-gated dead plugin-metric direct insert [#3788] #3812

mfreeman451 referenced this issue from a commit 2026-06-14 00:48:43 +00:00

chore(observability): remove flag-gated dead plugin-metric direct insert [#3788]

mfreeman451 referenced this issue from a commit 2026-06-14 00:56:50 +00:00

fix(agent-gateway): stop double-forwarding sysmon/snmp metric statuses [#3788]

mfreeman451 referenced this issue 2026-06-14 00:57:14 +00:00

fix(agent-gateway): stop double-forwarding sysmon/snmp metric statuses [#3788] #3813

mfreeman451 referenced this issue from a commit 2026-06-14 01:02:27 +00:00

refactor(core): remove dead sysmon/snmp metric-cutover no-op handlers [#3788]

mfreeman451 referenced this issue 2026-06-14 01:02:42 +00:00

refactor(core): remove dead sysmon/snmp metric-cutover no-op handlers [#3788] #3814

mfreeman451 referenced this issue from a commit 2026-06-14 01:05:44 +00:00

chore(agent-gateway): collapse sysmon/snmp metrics shadow flags to single enable [#3788]

mfreeman451 referenced this issue 2026-06-14 01:05:56 +00:00

chore(agent-gateway): collapse sysmon/snmp metrics shadow flags to single enable [#3788] #3815

mfreeman451 commented 2026-06-14 01:11:22 +00:00

Author

Owner

Copy link

Implementation status ledger (additive contract work)

Shipped a stack of small, independently-reviewable PRs covering every DELETE-list item and every isolated recommendation. Each is base staging unless noted.

✅ Tech-debt DELETE list — complete (4/4)

item	PR
Remove serviceradar.sysmon.shadow.v1 acceptance (metrics.ex)	#3810
Remove flag-gated dead maybe_insert_metrics + :plugin_result_direct_metrics_enabled	#3812
Remove vestigial no-op router handlers (handle_sysmon_metrics/handle_snmp_metrics/acknowledge_metrics_cutover + status_handler dispatch)	#3814 (stacked on #3813)
Collapse AGENT_GATEWAY_{SYSMON,SNMP}_METRICS_SHADOW_ENABLED env + helm *MetricsShadowEnabled aliases to the single *_METRICS_ENABLED flag	#3815

✅ Recommendations — isolated items shipped

REC	what	PR
REC1	carry OTLP-grade v2 semantics (schema_version 1→2, kind/temporality/resource scaffolding) in the plugin metric envelope	#3805
REC3	record whether plugin metric_type was producer-declared vs regex-inferred	#3808
REC4	dedup the metrics JetStream stream via Nats-Msg-Id = ingress_id + duplicate_window	#3804
REC7a	shared go/pkg/metricpoint serviceradar.metric.v1 builder for producers	#3809
REC7e	make the gateway SNMP interface-metric allowlist configurable (was a 2-OID hardcode)	#3807
REC9	reject mis-bucketed metric/OCSF payloads (guardrails)	#3803
REC10	surface silent metric_type→gauge coercion + dropped non-numeric plugin metrics via telemetry	#3806, #3802

✅ The "double work" / re-publish smell (TL;DR defect #1)

what	PR
Gateway stops forwarding sysmon-metrics/snmp-metrics statuses to core (the forward only hit the acknowledge_metrics_cutover no-op) — publishes directly to JetStream instead	#3813

⏳ Remaining RECs — owned elsewhere or design-gated (NOT started, by design)

REC	why not done here
REC2 (first-class metric emit in both WASM SDKs)	Actively owned by the add-service-monitoring-sdk-parity effort in serviceradar-sdk-go / serviceradar-sdk-rust — both repos already carry feat: add first-class telemetry emission commits + uncommitted work. Touching them would collide.
REC5 (single point-identity recipe; converge otel_metric_points recipe-v2 vs TimeseriesSeriesKey PKs)	Dual-table PK migration gated on open question #4 (which recipe wins). Needs an explicit decision + sequenced migration, not a small PR.
REC6 (kind/temporality-aware anomaly consumer; rate counters; histogram p50/p95)	anomaly_detection/** is the concurrent agent's domain — landed as the compact evaluator (#3792) and counter/cgroup specs (#3793, #3789/#3790). Counter normalization already shipped in #3795.
REC7b (migrate agent sysmon/snmp/icmp/mtr/sweep producers onto metricpoint)	Ordered LAST for good reason: depends on REC2 (SDK shape), REC5 (point identity), and REC6 (consumer). It's the cross-cutting COLLAPSE that rewires agent→gateway→core→anomaly atomically — not isolatable into a small reviewable PR until the above land.
REC8 (addon raw-OTLP one-lane vs two-lane)	Pure design decision (open question #2): retire otel.metrics.raw via a decode-and-republish shim, or keep it as the high-fidelity express lane.

Net: the additive contract groundwork (REC1/3/4/7a/7e/9/10), all migration-era debt deletion, and the double-forward fix are done and verified. What remains is the coordinated producer migration (REC7b) which is intentionally blocked behind the SDK parity effort (REC2), the point-identity decision (REC5), and the anomaly-consumer upgrade (REC6) — all either in flight on other branches or awaiting an open-question decision.

## Implementation status ledger (additive contract work) Shipped a stack of small, independently-reviewable PRs covering every DELETE-list item and every isolated recommendation. Each is base `staging` unless noted. ### ✅ Tech-debt DELETE list — complete (4/4) | item | PR | |---|---| | Remove `serviceradar.sysmon.shadow.v1` acceptance (`metrics.ex`) | **#3810** | | Remove flag-gated dead `maybe_insert_metrics` + `:plugin_result_direct_metrics_enabled` | **#3812** | | Remove vestigial no-op router handlers (`handle_sysmon_metrics`/`handle_snmp_metrics`/`acknowledge_metrics_cutover` + `status_handler` dispatch) | **#3814** *(stacked on #3813)* | | Collapse `AGENT_GATEWAY_{SYSMON,SNMP}_METRICS_SHADOW_ENABLED` env + helm `*MetricsShadowEnabled` aliases to the single `*_METRICS_ENABLED` flag | **#3815** | ### ✅ Recommendations — isolated items shipped | REC | what | PR | |---|---|---| | REC1 | carry OTLP-grade v2 semantics (`schema_version 1→2`, kind/temporality/resource scaffolding) in the plugin metric envelope | **#3805** | | REC3 | record whether plugin `metric_type` was producer-declared vs regex-inferred | **#3808** | | REC4 | dedup the `metrics` JetStream stream via `Nats-Msg-Id` = `ingress_id` + `duplicate_window` | **#3804** | | REC7a | shared `go/pkg/metricpoint` `serviceradar.metric.v1` builder for producers | **#3809** | | REC7e | make the gateway SNMP interface-metric allowlist configurable (was a 2-OID hardcode) | **#3807** | | REC9 | reject mis-bucketed metric/OCSF payloads (guardrails) | **#3803** | | REC10 | surface silent `metric_type→gauge` coercion + dropped non-numeric plugin metrics via telemetry | **#3806**, **#3802** | ### ✅ The "double work" / re-publish smell (TL;DR defect #1) | what | PR | |---|---| | Gateway stops forwarding `sysmon-metrics`/`snmp-metrics` statuses to core (the forward only hit the `acknowledge_metrics_cutover` no-op) — publishes directly to JetStream instead | **#3813** | ### ⏳ Remaining RECs — owned elsewhere or design-gated (NOT started, by design) | REC | why not done here | |---|---| | **REC2** (first-class metric emit in both WASM SDKs) | **Actively owned** by the `add-service-monitoring-sdk-parity` effort in `serviceradar-sdk-go` / `serviceradar-sdk-rust` — both repos already carry `feat: add first-class telemetry emission` commits + uncommitted work. Touching them would collide. | | **REC5** (single point-identity recipe; converge `otel_metric_points` recipe-v2 vs `TimeseriesSeriesKey` PKs) | Dual-table PK migration gated on **open question #4** (which recipe wins). Needs an explicit decision + sequenced migration, not a small PR. | | **REC6** (kind/temporality-aware anomaly consumer; rate counters; histogram p50/p95) | `anomaly_detection/**` is the concurrent agent's domain — landed as the compact evaluator (#3792) and counter/cgroup specs (#3793, #3789/#3790). Counter normalization already shipped in #3795. | | **REC7b** (migrate agent sysmon/snmp/icmp/mtr/sweep producers onto `metricpoint`) | Ordered **LAST** for good reason: depends on REC2 (SDK shape), REC5 (point identity), and REC6 (consumer). It's the cross-cutting COLLAPSE that rewires agent→gateway→core→anomaly atomically — not isolatable into a small reviewable PR until the above land. | | **REC8** (addon raw-OTLP one-lane vs two-lane) | Pure design decision (**open question #2**): retire `otel.metrics.raw` via a decode-and-republish shim, or keep it as the high-fidelity express lane. | **Net:** the additive contract groundwork (REC1/3/4/7a/7e/9/10), all migration-era debt deletion, and the double-forward fix are done and verified. What remains is the coordinated producer migration (REC7b) which is intentionally blocked behind the SDK parity effort (REC2), the point-identity decision (REC5), and the anomaly-consumer upgrade (REC6) — all either in flight on other branches or awaiting an open-question decision.

mfreeman451 commented 2026-06-14 01:11:31 +00:00

Author

Owner

Copy link

Implementation status ledger (additive contract work)

Shipped a stack of small, independently-reviewable PRs covering every DELETE-list item and every isolated recommendation. Each is base staging unless noted.

✅ Tech-debt DELETE list — complete (4/4)

item	PR
Remove serviceradar.sysmon.shadow.v1 acceptance (metrics.ex)	#3810
Remove flag-gated dead maybe_insert_metrics + :plugin_result_direct_metrics_enabled	#3812
Remove vestigial no-op router handlers (handle_sysmon_metrics/handle_snmp_metrics/acknowledge_metrics_cutover + status_handler dispatch)	#3814 (stacked on #3813)
Collapse AGENT_GATEWAY_{SYSMON,SNMP}_METRICS_SHADOW_ENABLED env + helm *MetricsShadowEnabled aliases to the single *_METRICS_ENABLED flag	#3815

✅ Recommendations — isolated items shipped

REC	what	PR
REC1	carry OTLP-grade v2 semantics (schema_version 1→2, kind/temporality/resource scaffolding) in the plugin metric envelope	#3805
REC3	record whether plugin metric_type was producer-declared vs regex-inferred	#3808
REC4	dedup the metrics JetStream stream via Nats-Msg-Id = ingress_id + duplicate_window	#3804
REC7a	shared go/pkg/metricpoint serviceradar.metric.v1 builder for producers	#3809
REC7e	make the gateway SNMP interface-metric allowlist configurable (was a 2-OID hardcode)	#3807
REC9	reject mis-bucketed metric/OCSF payloads (guardrails)	#3803
REC10	surface silent metric_type→gauge coercion + dropped non-numeric plugin metrics via telemetry	#3806, #3802

✅ The "double work" / re-publish smell (TL;DR defect #1)

what	PR
Gateway stops forwarding sysmon-metrics/snmp-metrics statuses to core (the forward only hit the acknowledge_metrics_cutover no-op) — publishes directly to JetStream instead	#3813

⏳ Remaining RECs — owned elsewhere or design-gated (NOT started, by design)

REC	why not done here
REC2 (first-class metric emit in both WASM SDKs)	Actively owned by the add-service-monitoring-sdk-parity effort in serviceradar-sdk-go / serviceradar-sdk-rust — both repos already carry feat: add first-class telemetry emission commits + uncommitted work. Touching them would collide.
REC5 (single point-identity recipe; converge otel_metric_points recipe-v2 vs TimeseriesSeriesKey PKs)	Dual-table PK migration gated on open question #4 (which recipe wins). Needs an explicit decision + sequenced migration, not a small PR.
REC6 (kind/temporality-aware anomaly consumer; rate counters; histogram p50/p95)	anomaly_detection/** is the concurrent agent's domain — landed as the compact evaluator (#3792) and counter/cgroup specs (#3793, #3789/#3790). Counter normalization already shipped in #3795.
REC7b (migrate agent sysmon/snmp/icmp/mtr/sweep producers onto metricpoint)	Ordered LAST for good reason: depends on REC2 (SDK shape), REC5 (point identity), and REC6 (consumer). It's the cross-cutting COLLAPSE that rewires agent→gateway→core→anomaly atomically — not isolatable into a small reviewable PR until the above land.
REC8 (addon raw-OTLP one-lane vs two-lane)	Pure design decision (open question #2): retire otel.metrics.raw via a decode-and-republish shim, or keep it as the high-fidelity express lane.

Net: the additive contract groundwork (REC1/3/4/7a/7e/9/10), all migration-era debt deletion, and the double-forward fix are done and verified. What remains is the coordinated producer migration (REC7b) which is intentionally blocked behind the SDK parity effort (REC2), the point-identity decision (REC5), and the anomaly-consumer upgrade (REC6) — all either in flight on other branches or awaiting an open-question decision.

## Implementation status ledger (additive contract work) Shipped a stack of small, independently-reviewable PRs covering every DELETE-list item and every isolated recommendation. Each is base `staging` unless noted. ### ✅ Tech-debt DELETE list — complete (4/4) | item | PR | |---|---| | Remove `serviceradar.sysmon.shadow.v1` acceptance (`metrics.ex`) | **#3810** | | Remove flag-gated dead `maybe_insert_metrics` + `:plugin_result_direct_metrics_enabled` | **#3812** | | Remove vestigial no-op router handlers (`handle_sysmon_metrics`/`handle_snmp_metrics`/`acknowledge_metrics_cutover` + `status_handler` dispatch) | **#3814** *(stacked on #3813)* | | Collapse `AGENT_GATEWAY_{SYSMON,SNMP}_METRICS_SHADOW_ENABLED` env + helm `*MetricsShadowEnabled` aliases to the single `*_METRICS_ENABLED` flag | **#3815** | ### ✅ Recommendations — isolated items shipped | REC | what | PR | |---|---|---| | REC1 | carry OTLP-grade v2 semantics (`schema_version 1→2`, kind/temporality/resource scaffolding) in the plugin metric envelope | **#3805** | | REC3 | record whether plugin `metric_type` was producer-declared vs regex-inferred | **#3808** | | REC4 | dedup the `metrics` JetStream stream via `Nats-Msg-Id` = `ingress_id` + `duplicate_window` | **#3804** | | REC7a | shared `go/pkg/metricpoint` `serviceradar.metric.v1` builder for producers | **#3809** | | REC7e | make the gateway SNMP interface-metric allowlist configurable (was a 2-OID hardcode) | **#3807** | | REC9 | reject mis-bucketed metric/OCSF payloads (guardrails) | **#3803** | | REC10 | surface silent `metric_type→gauge` coercion + dropped non-numeric plugin metrics via telemetry | **#3806**, **#3802** | ### ✅ The "double work" / re-publish smell (TL;DR defect #1) | what | PR | |---|---| | Gateway stops forwarding `sysmon-metrics`/`snmp-metrics` statuses to core (the forward only hit the `acknowledge_metrics_cutover` no-op) — publishes directly to JetStream instead | **#3813** | ### ⏳ Remaining RECs — owned elsewhere or design-gated (NOT started, by design) | REC | why not done here | |---|---| | **REC2** (first-class metric emit in both WASM SDKs) | **Actively owned** by the `add-service-monitoring-sdk-parity` effort in `serviceradar-sdk-go` / `serviceradar-sdk-rust` — both repos already carry `feat: add first-class telemetry emission` commits + uncommitted work. Touching them would collide. | | **REC5** (single point-identity recipe; converge `otel_metric_points` recipe-v2 vs `TimeseriesSeriesKey` PKs) | Dual-table PK migration gated on **open question #4** (which recipe wins). Needs an explicit decision + sequenced migration, not a small PR. | | **REC6** (kind/temporality-aware anomaly consumer; rate counters; histogram p50/p95) | `anomaly_detection/**` is the concurrent agent's domain — landed as the compact evaluator (#3792) and counter/cgroup specs (#3793, #3789/#3790). Counter normalization already shipped in #3795. | | **REC7b** (migrate agent sysmon/snmp/icmp/mtr/sweep producers onto `metricpoint`) | Ordered **LAST** for good reason: depends on REC2 (SDK shape), REC5 (point identity), and REC6 (consumer). It's the cross-cutting COLLAPSE that rewires agent→gateway→core→anomaly atomically — not isolatable into a small reviewable PR until the above land. | | **REC8** (addon raw-OTLP one-lane vs two-lane) | Pure design decision (**open question #2**): retire `otel.metrics.raw` via a decode-and-republish shim, or keep it as the high-fidelity express lane. | **Net:** the additive contract groundwork (REC1/3/4/7a/7e/9/10), all migration-era debt deletion, and the double-forward fix are done and verified. What remains is the coordinated producer migration (REC7b) which is intentionally blocked behind the SDK parity effort (REC2), the point-identity decision (REC5), and the anomaly-consumer upgrade (REC6) — all either in flight on other branches or awaiting an open-question decision.

mfreeman451 referenced this issue from a commit 2026-06-14 02:57:05 +00:00

feat(metricpoint): add shared serviceradar.metric.v1 builder for producers [#3788 REC7]

mfreeman451 referenced this issue from a commit 2026-06-14 02:58:29 +00:00

feat(metrics): carry OTLP-grade v2 semantics in plugin metric envelope [#3788 REC1]

mfreeman451 referenced this issue from a commit 2026-06-14 03:01:13 +00:00

feat(snmp): make the gateway interface-metric allowlist configurable [#3788 REC7e]

mfreeman451 referenced this issue from a commit 2026-06-14 03:03:37 +00:00

chore(observability): remove flag-gated dead plugin-metric direct insert [#3788]

mfreeman451 referenced this issue from a commit 2026-06-14 03:04:18 +00:00

chore(agent-gateway): collapse sysmon/snmp metrics shadow flags to single enable [#3788]

mfreeman451 referenced this issue from a commit 2026-06-14 03:06:51 +00:00

feat(metrics): record whether plugin metric_type was inferred [#3788 REC3]

mfreeman451 referenced this issue from a commit 2026-06-14 03:25:27 +00:00

feat(core): reject metric bodies mislabeled as OCSF events at the source [#3788 REC10]

mfreeman451 referenced this issue 2026-06-14 03:25:45 +00:00

feat(core): reject metric bodies mislabeled as OCSF events at the source [#3788 REC10] #3823

mfreeman451 referenced this issue from a commit 2026-06-14 03:37:29 +00:00

Merge pull request 'chore(metrics): remove dead serviceradar.sysmon.shadow.v1 acceptance [#3788]' (#3810) from work/3788-remove-sysmon-shadow-schema into staging

mfreeman451 referenced this issue from a commit 2026-06-14 03:39:08 +00:00

Merge pull request 'feat(metrics): dedup the metrics JetStream stream via Nats-Msg-Id [#3788 REC4]' (#3804) from work/3788-rec4-metrics-stream-dedup into staging

mfreeman451 referenced this issue from a commit 2026-06-14 03:39:26 +00:00

Merge pull request 'feat(event-writer): surface silent metric_type->gauge coercion [#3788 REC10]' (#3806) from work/3788-rec10-consumer-metric-type-telemetry into staging

mfreeman451 referenced this issue from a commit 2026-06-14 03:41:09 +00:00

Merge pull request 'feat(event-writer): reject mis-bucketed metric/OCSF payloads [#3788 REC9]' (#3803) from work/3788-rec9-ocsf-misbucket-guardrails into staging

mfreeman451 referenced this issue from a commit 2026-06-14 03:41:28 +00:00

Merge pull request 'chore(observability): remove flag-gated dead plugin-metric direct insert [#3788]' (#3812) from work/3788-remove-plugin-direct-metric-insert into staging

mfreeman451 referenced this issue from a commit 2026-06-14 03:41:53 +00:00

Merge pull request 'feat(core): reject metric bodies mislabeled as OCSF events at the source [#3788 REC10]' (#3823) from work/3788-rec10-reject-misbucketed-ocsf-source into staging

mfreeman451 referenced this issue from a commit 2026-06-14 03:42:42 +00:00

Merge pull request 'feat(metrics): carry OTLP-grade v2 semantics in plugin metric envelope [#3788 REC1]' (#3805) from work/3788-rec1-plugin-metric-v2-envelope into staging

mfreeman451 referenced this issue from a commit 2026-06-14 03:44:18 +00:00

feat(metrics): record whether plugin metric_type was inferred [#3788 REC3]

mfreeman451 referenced this issue 2026-06-14 03:44:20 +00:00

feat(metrics): record whether plugin metric_type was inferred [#3788 REC3] #3826

mfreeman451 referenced this issue from a commit 2026-06-14 03:44:39 +00:00

Merge pull request 'feat(metrics): record whether plugin metric_type was inferred [#3788 REC3]' (#3826) from work/3788-rec3-record-inferred-metric-type into staging

mfreeman451 referenced this issue from a commit 2026-06-14 03:47:22 +00:00

fix(agent-gateway): surface dropped plugin metrics via telemetry

mfreeman451 referenced this issue from a commit 2026-06-14 03:47:43 +00:00

Merge pull request 'fix(agent-gateway): surface dropped plugin metrics via telemetry [#3788 REC10]' (#3802) from work/3788-rec10-surface-plugin-metric-drops into staging

mfreeman451 referenced this issue from a commit 2026-06-14 03:48:29 +00:00

Merge pull request 'chore(agent-gateway): collapse sysmon/snmp metrics shadow flags to single enable [#3788]' (#3815) from work/3788-collapse-metrics-shadow-flags into staging

mfreeman451 referenced this issue from a commit 2026-06-14 03:49:04 +00:00

Merge pull request 'feat(snmp): make the gateway interface-metric allowlist configurable [#3788 REC7e]' (#3807) from work/3788-rec7e-snmp-configurable-oid-allowlist into staging

mfreeman451 referenced this issue from a commit 2026-06-14 03:49:23 +00:00

Merge pull request 'fix(agent-gateway): stop double-forwarding sysmon/snmp metric statuses [#3788]' (#3813) from work/3788-skip-redundant-metric-forward into staging

mfreeman451 referenced this issue from a commit 2026-06-14 03:50:05 +00:00

Merge pull request 'refactor(core): remove dead sysmon/snmp metric-cutover no-op handlers [#3788]' (#3814) from work/3788-remove-dead-metric-handlers into staging

mfreeman451 referenced this issue from a commit 2026-06-14 03:50:54 +00:00

Merge pull request 'feat(metricpoint): shared serviceradar.metric.v1 builder for producers [#3788 REC7]' (#3809) from work/3788-rec7a-metricpoint-builder into staging

mfreeman451 referenced this issue 2026-06-14 18:39:57 +00:00

feat(anomaly): protobuf metric envelope cutover + #3796 engine hardening #3829

mfreeman451 referenced this issue 2026-06-29 14:35:36 +00:00

Remove legacy sysmon shadow/cutover metric tech-debt (post-JetStream migration) [#3788] #3801

mfreeman451 referenced this issue 2026-06-29 14:36:52 +00:00

Remove legacy sysmon shadow/cutover metric tech-debt (post-JetStream migration) [#3788] #3801

mfreeman451 commented 2026-06-29 14:53:45 +00:00

Author

Owner

Copy link

Re-scoped against current staging + the OpenSpec set: #3788 is already decomposed and partly landed — it does not need a fresh from-scratch implementation. A proposal drafted for this pass surfaced that the work has moved well past this issue's snapshot.

Already decomposed across existing OpenSpec changes

• add-protobuf-metric-envelope — ✓ Complete (landed). The binary serviceradar.metric.v1 wire format ships: elixir/serviceradar_core/lib/serviceradar/proto/metric/v1/metric.pb.ex + observability/metric_envelope.ex (its device_resolver fixes the per-device collapse). This is a hard cutover.
• update-anomaly-evaluation-cadence (7/25 tasks). Its "Decision 0" already defines the serviceradar.metric.v1 field contract and ADDs Canonical Metric Signal Contract / First-Class Metric Path / Metric Stream Idempotency / Metric Schema Guardrails to the ingestion-routing capability — the largest overlap with this issue.
• add-monotonic-counter-metric-semantics (8/16 tasks). Owns the kind/temporality/counter slice — the "a cumulative counter fed raw into z-sigma looks like a ramp" defect.
• Downstream consumers: add-cgroup-v2-tenant-metrics (0/15), add-delta-metrics-lakehouse (1/22). Adjacent: add-bulk-payload-pipeline (0/21 — the gateway forward path).

The real open question is architectural, not "go implement #3788"

There is a philosophy clash between the additive schema_version 1→2 path described here (and in update-anomaly-evaluation-cadence) and the hard cutover already shipped in add-protobuf-metric-envelope. That needs a human decision before any further metric-contract work — landing a 4th parallel effort would only deepen the conflict.

Still genuinely open (regardless of which philosophy wins)

• Finishing update-anomaly-evaluation-cadence + add-monotonic-counter-metric-semantics.
• The cross-repo producer emit paths — first-class metric emit in ~/src/serviceradar-sdk-go, ~/src/serviceradar-sdk-rust, and the Go agent (sysmon/snmp/icmp/mtr/sweep/rperf), incl. the currently-dropped sysmon Network[] and SNMP non-octet OIDs.
• The series-identity unification + single-sink migration (the 3-sinks → 1 part: cpu_metrics / timeseries_metrics / otel_metric_points).
• The legacy sysmon/snmp reject-handler removal (the tail of #3801).

Recommendation

Treat #3788 as the umbrella that has been decomposed — either close it linking the changes above, or keep it as the tracking umbrella, but do not open a 4th parallel proposal. The next concrete step is the human consolidation call (additive-vs-cutover) plus finishing the two in-flight changes. (A consolidation-mapping draft exists locally and can be attached if useful.)

Re-scoped against current `staging` + the OpenSpec set: **#3788 is already decomposed and partly landed — it does not need a fresh from-scratch implementation.** A proposal drafted for this pass surfaced that the work has moved well past this issue's snapshot. ### Already decomposed across existing OpenSpec changes - **`add-protobuf-metric-envelope` — ✓ Complete (landed).** The binary `serviceradar.metric.v1` wire format ships: `elixir/serviceradar_core/lib/serviceradar/proto/metric/v1/metric.pb.ex` + `observability/metric_envelope.ex` (its `device_resolver` fixes the per-device collapse). This is a **hard cutover**. - **`update-anomaly-evaluation-cadence` (7/25 tasks).** Its "Decision 0" already defines the `serviceradar.metric.v1` field contract and ADDs `Canonical Metric Signal Contract` / `First-Class Metric Path` / `Metric Stream Idempotency` / `Metric Schema Guardrails` to the `ingestion-routing` capability — the largest overlap with this issue. - **`add-monotonic-counter-metric-semantics` (8/16 tasks).** Owns the `kind`/`temporality`/counter slice — the "a cumulative counter fed raw into z-sigma looks like a ramp" defect. - Downstream consumers: `add-cgroup-v2-tenant-metrics` (0/15), `add-delta-metrics-lakehouse` (1/22). Adjacent: `add-bulk-payload-pipeline` (0/21 — the gateway forward path). ### The real open question is architectural, not "go implement #3788" There is a philosophy clash between the **additive `schema_version 1→2`** path described here (and in `update-anomaly-evaluation-cadence`) and the **hard cutover** already shipped in `add-protobuf-metric-envelope`. That needs a human decision before any further metric-contract work — landing a 4th parallel effort would only deepen the conflict. ### Still genuinely open (regardless of which philosophy wins) - Finishing `update-anomaly-evaluation-cadence` + `add-monotonic-counter-metric-semantics`. - The **cross-repo producer emit paths** — first-class metric emit in `~/src/serviceradar-sdk-go`, `~/src/serviceradar-sdk-rust`, and the Go agent (sysmon/snmp/icmp/mtr/sweep/rperf), incl. the currently-dropped sysmon `Network[]` and SNMP non-octet OIDs. - The series-identity unification + single-sink migration (the 3-sinks → 1 part: `cpu_metrics` / `timeseries_metrics` / `otel_metric_points`). - The legacy sysmon/snmp reject-handler removal (the tail of #3801). ### Recommendation Treat #3788 as the umbrella that has been decomposed — either **close it** linking the changes above, or keep it as the tracking umbrella, but do **not** open a 4th parallel proposal. The next concrete step is the human consolidation call (**additive-vs-cutover**) plus finishing the two in-flight changes. (A consolidation-mapping draft exists locally and can be attached if useful.)

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

staging

docs/agents-dialyzer-no-workarounds

revert/dialyzer-pr-4677

fix/dialyzer-core-4676

add-sweep-profile-mtr

feat/awx-live-launch-preflight

codex/issue-4617-awx-live-launch-preflight

demo/prod-release

unify-sweep-results-proto

adopt-ash-codegen-migrations

main

consolidate-serviceradar-cli

codex/fix-web-ng-dialyzer

codex/issue-4634-native-addon-rollouts

codex/4652-seasonal-disposition-permissions

fix/addon-backfill-oban-state-cast

codex/fix-extension-release-selection

fix/awx-target-hold-not-found

codex/fix-release-retry-registry-digest

fix/awx-launch-result-sanitizer

codex/fix-release-retry-digest

codex/release-v1.4.23

codex/fix-self-scheduling-oban-workers

docs/remote-access-spec-reconciliation

feat/cold-storage-tiering-proposal

codex/plugin-recovery-db-validation-wip

fix/native-addon-import-v1.4.22

codex/anomaly-4604-regression-gaps

agent/control-plane-runtime-listener-1.4.20

fix/release-external-artifact-pins

release/v1.4.19

fix/staging-rust-release-inputs

bazel-rust-fix-v2

chart/serviceradar-1.4.18-smtp-secret-refs

release/v1.4.16-refresh

proposal/proxmox-native-guest-terminal-console

fix/arancini-lib-0.7.3

fix/srql-fixture-logs-source-ip

release/v1.4.16

bazel-rust-fix

demo/remote-access-a8e139d7-rollout

fix/awx-restricted-marker-survey

fix/awx-callback-binding-contract

feat/secure-callback-orchestrator

causal-engine

cargo-build-fix

release/v1.4.15

feat/secure-remote-access

fix/sync-ingest-addon-import

fix/demo-v1-4-14-release-source

release/v1.4.14

feat/automation-callback-http-authority

feat/awx-ephemeral-callback-credential

feat/automation-launch-envelope

feat/automation-callback-grants

openspec/add-proxmox-qemu-graphical-console

openspec/fix-proxmox-identity-terminal-console

openspec/enable-agent-routed-ssh

feat/harden-ansible-awx-targeting

openspec/harden-ansible-awx-targeting

openspec/publish-ssh-ca-ansible-enrollment

openspec/add-automation-callback-grants

docs/archive-awx-playbook-execution

docs/archive-awx-inventory-sync

docs/reconcile-remote-access-foundation

fix/plugin-result-handler-failures

release/v1.4.12

fix/scalibr-endpoint-inventory-012

fix/web-nats-ingest-acl

proposal/scale-addon-fleet-view

fix/fatal-multivalue-topn

fix/workload-identity-spool

fix/native-addon-sync-existing

fix/nats-reconnect-test-flake

fix/core-ci-flakes

fix/log-promotion-uuid-metadata

release/v1.4.11

fix/v1.4.11-release-blockers

fix/wasm-tinygo-gomodcache-isolation

updates/crypto-vuln

renovate/registry.carverauto.dev-serviceradar-forgejo-ci

openspec/remediate-armis-overmerge-disposition

fix/web-ng-precommit-staging

fix/armis-switchport-metadata

hosted-otlp-managed-dns-toggle

fix-rperf-chart-toggle

fix-cnpg-linode-barman-env

release/v1.4.8

fix/agent-release-retry-and-403

fix/device-metadata-false-integration-provenance

fix/ansible-run-primary-read-crash

fix/topology-refresh-timeout

fix/awx-ingest-device-ip

fix/plugin-assignment-version-dropdown

fix/netflow-popup-body-overlay

release/v1.4.7

fix/v147-gate-playbook

ci/enforce-baseline-gate

fix/run-task-awx-applicability

fix/device-panel-launchable-playbooks

feat/srql-awx-managed-field

fix/regenerate-schema-baseline

fix/startup-blocking-migrations

fix/trivy-scan-event-noise

fix/admin-openapi-and-schema-baseline

fix/elixir-quality-staging

ci/auto-publish-wasm-plugins-on-release

fix/srql-monaco-metadata-validation

release/v1.4.6

fix/otx-memory-256

fix/otx-plugin-version-reconcile

fix/v146-gate-format-lint

feat/event-details-device-link

fix/apikey-delete-and-apidocs-auth

feat/device-detail-ansible-panel

fix/device-details-discovery-sources-compact

feat/metadata-srql-search

fix/proxmox-pressure-event-hysteresis

fix/credential-resolution-event-noise

fix/config-health-flap-stale-push

fix/srql-autocomplete

fix/netflow-local-cidr-map-anchors

feat/publish-openapi-spec

fix/netflow-flowpath-popup-clip

fix/awx-inventory-sync-invalid-signature

fix/logs-source-filter-index

fix/topbar-dropdown-zindex

fix/srql-force-custom-plan

fix/logs-index-migration-online-safe

release/v1.4.5

fix/source-authoritative-device-identity

chore/v145-format-lint-cleanup

fix/observability-severity-catalog-and-index

fix/camera-relay-stream-upgrade-session-cookie

fix/dusk-localhost-scope-and-narrow

fix/settings-nav-collapse-state

fix/alienvault-otx-daily-backoff

feat/device-details-all-metadata

test/api-endpoint-suite

fix/log-and-trace-noise

feat/profile-menu-and-api-docs

fix/api-device-list-pagination

renovate/node_20_alpine-20-alpine

release/v1.4.4

fix/ansible-retention-no-primary-read

fix/proxmox-identity-consolidation

fix/sysmon-resolution-and-merge-continuity

fix/device-details-all-integrations

fix/oauth-api-access

bug/startup-migrations-normalize-ownership

fix/credential-rules-db-surface

fix/alienvault-otx-flakiness

release/v1.4.3

codex/fix-armis-northbound-orphaned-oban

release/v1.4.2

perf/decouple-config-hash-credential-resolution

feat/awx-inventory-sync-hardening

codex/awx-inventory-sync-deployment-4422

codex/fix-dire-multihomed-pve-4416

codex/fix-credential-rule-form-4415

codex/fix-unifi-protect-4420

codex/fix-proxmox-inventory-4417-4419

docs/awx-inventory-sync-proposal

release/v1.4.1

codex/harden-auth-defaults

worktree-anomaly-engine-overhaul

add-cnpg-backup-values

renovate/ubuntu_jammy-22.04

feat/settings-catalog-finalize

feat/settings-catalog-phase3

feat/settings-catalog-phase2

chore/web-ng-mix-format

feat/settings-catalog-shell-phase1

feat/camera-plugins-envelope-host

feat/camera-credential-materializer-elixir

docs/openspec-settings-catalog-redesign

fix/netprobe-attach-health-race

fix/scalibr-addon-defaults

fix/addon-override-collapse

fix/addon-profile-target-agents

fix/srql-comment-semicolon

fix/agent-release-status-message

fix/anomaly-index-comment

fix/oidc-login-user-arg

release/v1.3.10

feat/sso-gate-jit-provisioning

fix/dup-migration-version-v139

fix/oci-legacy-signature-idempotent

release/v1.3.9

fix/anomaly-confirmation-gate

fix/syslog-severity-zen-declobber

feat/auth-local-login-per-account-and-breakglass

fix/oidc-token-verify-hardening

fix/current-password-bcrypt-2y

fix/oidc-settings-entra-preset

fix/web-ng-event-details-log-name-overflow

fix/oidc-session-samesite-lax

fix/auth-settings-oidc-secret-cloak

fix/auth-password-and-profile

fix/plugin-camera-host-required

fix/snmp-64bit-selector

fix/leader-node-card-overflow

fix/snmp-counter32-wrap

fix/interface-charts-ux

fix/oban-self-reschedule-churn

fix/plugin-wasm-mirror-and-retention

fix/agent-command-bus-rpc-fallback

fix/addon-package-accept-resources

release/v1.3.8

fix/flow-attribution-prune-batched

perf/churn-table-tuning

perf/srql-cagg-routing

perf/workload-identity-skip-guard

fix/agent-config-version-wedge

fix/nats-logs-subject-overlap

fix/elixir-dep-security

refactor/anomaly-engine-rigor

docs/deepcausality-anomaly-paper

release/v1.3.7

mf/anomaly-semantics-overlay-integration

mf/anomaly-semantics-and-capacity-sanity

fix/snmp-anomaly-target-attribution

add-device-chart-anomaly-overlays

fix/device-cpu-graphs

fix/release-tag-commit-publish

fix/release-retry-image-push

release/v1.3.6

mf/fix-core-alert-netflow-errors

mf/topology-shared-skip-guard

mf/fix-causal-prediction-uuid

mf/land-mega-sprint

backup-web-ng-stack

mf/fix-topology-guard

mf/retire-uasb-code

mf/retire-uasb

mf/addon-modularize

mf/align-anomaly-series-key

mf/addon-snmp-target-from-tags

mf/auto-resolve-stale-anomalies

mf/perf-topology-skip-unchanged-report

mf/fix-mapper-resolve-local-ifname

mf/fix-mapper-canonical-mac-portid

mf/fix-interface-identity-mac-normalization

mf/perf-topology-conditional-canonical-set

mf/perf-topology-skip-unchanged-rebuild

mf/fix-cnpg-demo-storage-size

mf/fix-scan-linux-bpf-unix-import

mf/agent2-50-1-device-live-index-split

mf/agent2-50-1-topology-graph-split

mf/agent2-50-1-networks-live-split

mf/agent2-37-1-netflow-interface-scope

mf/agent2-28-3-chart-focus-tracking

mf/agent1-51-8-native-addon-gates

mf/agent1-51-7-flow-collector-verify

mf/agent1-51-4-core-elx-verify

mf/agent2-50-2-srql-devices-filters

mf/agent2-50-2-srql-downsample

mf/agent2-50-2-srql-events

mf/agent2-50-2-srql-interfaces

mf/agent2-50-2-srql-query-mod

mf/agent1-50-1-syn-scanner-split

mf/agent2-50-2-srql-parser

mf/agent2-28-2-threshold-lines

mf/agent1-50-1-sweeper-split

mf/agent2-27-1-counter-width

mf/agent2-30-2-netflow-empty-states

mf/agent2-31-2-stacked100-volume

mf/agent2-30-1-chart-null-gaps

mf/agent2-28-1-timeseries-annotations

mf/agent2-45-2-topology-parallel-links

chore/uncheck-31-1-process-count-chart

mf/agent2-45-3-topology

mf/agent2-36-1-table-pagination

mf/agent2-32-timeseries-modularization

mf/agent2-36-2-topology-cap

fix/flow-sampling-rate-e2e

fix/recover-anomaly-alerting

fix/device-findings-canonical-scope

fix/device-panel-actionable-rows

fix/sysmon-debug-spike-smoke

fix/device-panel-rendered-field-projection

fix/topology-runtime-projection

chore/mark-54-3-observability-cpu-complete

fix/device-panel-anomaly-capacity-lookups

fix/log-insert-placeholders

fix/dire-lookup-indexes

fix/log-promotion-rule-cache

fix/inventory-rollup-small-batches

fix/flow-enrichment-provider-cache

fix/recover-seasonal-profile-state

chore/reconcile-tasks-4096-4097

fix/endpoint-inventory-async-results

fix/endpoint-inventory-agent-fairness

fix/endpoint-inventory-scan-lookup-index

fix/endpoint-inventory-pretransaction-noop

fix/endpoint-inventory-noop-short-circuit

fix/endpoint-inventory-cancel-timeout

fix/endpoint-inventory-inner-timeout

fix/anomaly-config-tuning-ownership

fix/srql-filter-hardening

fix/mtr-destination-hop-rtt

fix/mapper-ifxtable-names

fix/device-bulk-tag-scope

fix/unifi-pagination

fix/sweeper-syn-status-races

fix/recover-anomaly-perf-delivery

fix/status-handler-results-timeout

fix/recover-anomaly-verdict-parity-docs

fix/addon-grpc-stream-loss

fix/event-writer-terminal-delivery

fix/mapper-fdb-progress

fix/mapper-snmp-connect-once

fix/syn-scanner-port-attribution

fix/srql-cagg-bucket-bounds

fix/srql-interface-lateral-limit

fix/srql-flows-stats-unicode

fix/srql-null-negation

fix/srql-order-tiebreaks

fix/srql-discovery-sources-overlap

fix/srql-dos-hardening

fix/topology-cypher-escape

fix/sysmon-process-count-total

fix/sysmon-per-core-max-charts

fix/forgejo-ci-concurrency

fix/srql-timeseries-other-rollup

fix/topology-age-indexes

fix/topology-runtime-graph-cache

fix/netflow-cache-refresh-scan-window

catchup/netflow-sankey-other

fix/ci-action-flood

catchup/anomaly-3953-3954

docs/anomaly-poison-recovery-state-tradeoff

fix/flow-direction-count-null-semantics

fix/netflow-summary-covered-span

fix/netflow-interface-window-p95

fix/netflow-sankey-tail-other

fix/anomaly-scoring-liveness

perf/anomaly-counter-state-in-place

fix/anomaly-counter-wrap-metadata

proposal/fix-anomaly-engine-semantics-and-delivery

fix/anomaly-dashboard-query-safety

test/anomaly-dashboard-viewer-variable-safety

fix/anomaly-authored-kpi-trend-time

fix/anomaly-dashboard-recent-sparklines

fix/anomaly-authored-aggregate-full-results

fix/anomaly-edge-transition-findings

fix/anomaly-telemetry-broadcast-delivery

fix/anomaly-addon-config-validation

fix/anomaly-edge-state-cap-eviction

fix/anomaly-core-numeric-safety

fix/anomaly-feed-task-lifecycle

fix/anomaly-core-ingestion-fidelity

fix/anomaly-blank-numeric-params

fix/anomaly-blank-param-repair

fix/anomaly-addon-prompt-shutdown

fix/anomaly-addon-resource-limits

fix/anomaly-canonical-edge-identity

fix/anomaly-snmp-target-attribution

fix/anomaly-sanitize-edge-verdict-subjects

fix/anomaly-interface-capacity-partition

fix/anomaly-harden-series-identity

fix/anomaly-stabilize-central-verdict-ids

fix/anomaly-seasonal-bucket-window

fix/anomaly-seasonal-clears

fix/anomaly-seasonal-profile-column-gate

fix/anomaly-seasonal-enqueue-uniqueness

fix/anomaly-flow-capacity-units

fix/anomaly-seasonal-negative-slope-eta

fix/anomaly-capacity-wrap-gap

fix/anomaly-capacity-warning-horizon

fix/anomaly-confirm-slot-semantics

fix/anomaly-memory-override-class

fix/anomaly-window-duration-scope

test/anomaly-edge-verdict-sample-time

test/anomaly-series-key-parity

docs/anomaly-window-envelope

perf/anomaly-worker-history-pages

perf/anomaly-causal-bulk-inserts

fix/anomaly-seasonal-profile-timezone

feat/srql-seasonal-profile-hour-of-week

test/srql-seasonal-profile-integration

fix/anomaly-seasonal-state-persistence

fix/agent-addon-stream-reconnect

fix/agent-addon-circuit-cooldown

fix/addon-stream-loss-diagnostics

docs/anomaly-edge-tuning-ownership

fix/anomaly-capacity-alert-gates

fix/anomaly-alert-source-severity

fix/device-anomaly-capacity-query-scope

fix/anomaly-capacity-actionable-rows

fix/snmp-anomaly-status-bucket

fix/anomaly-capacity-friendly-labels

fix/chart-hover-geometry

fix/chart-hover-target-tooltips

fix/chart-gap-honesty

fix/chart-y-axis-grids

fix/timeseries-y-scale-data-band

fix/timeseries-row-units

fix/timeseries-counter-gap-rendering

fix/timeseries-counter-clamp-scope

fix/timeseries-measured-rate-spikes

fix/timeseries-envelope-downsampling

fix/topology-stable-node-ids

fix/timeseries-series-patterns

fix/dashboard-table-bounds

fix/sysmon-process-count-chart

fix/sysmon-process-sparklines

fix/netflow-stacked100-total-volume

fix/netflow-rate-normalized-summaries

fix/netflow-enrichment-expiry-chart-errors

fix/netflow-interface-gauge-scope

fix/netflow-canonical-conversations

fix/snmp-interface-srql-rates

fix/snmp-grouped-rate-series

fix/dashboard-table-sorting

fix/ocsf-events-retention-reconcile

fix/dashboard-vars-embedded-literals

fix/netflow-interface-p95-batch

fix/snmp-anomaly-subclass-bucket

fix/anomaly-capacity-task-timeouts

fix/dashboard-table-blanks-last

fix/interface-metric-first-key

fix/netflow-enrichment-expiry-helper

fix/anomaly-shutdown-checkpoint-flush

fix/anomaly-clear-hysteresis

fix/anomaly-cap-eviction-amortized

fix/anomaly-small-rate-floor

fix/anomaly-poison-preserve-state

fix/anomaly-finding-title-time-diagnostics

fix/causal-finding-row-convergence

fix/seasonal-anomaly-clear-contract

fix/flow-capacity-threshold-override

fix/anomaly-finding-dimension-title

fix/anomaly-addon-snmp-target-tags

fix/anomaly-addon-clear-confirmation-test

fix/web-ng-distinct-rate-units

feat/srql-flow-stats-other-rollup

fix/capacity-forecast-index-hypertable

fix/web-ng-enrichment-expiry-sql-parens

fix/core-hide-versioned-anomaly-series-labels

fix/netflow-summary-covered-span-floor

fix/netflow-summary-requested-window-rate

fix/timeseries-empty-states

fix/interface-metrics-empty-links

fix/timeseries-safe-error-empty-state

feat/timeseries-annotation-markers

feat/timeseries-reference-lines

feat/sysmon-series-max-charts

feat/sysmon-finding-chart-markers

fix/sysmon-finding-marker-single-series-test

feat/sysmon-threshold-reference-lines

fix/sysmon-saturation-gate-labels

test/sysmon-saturation-gate-parity

test/web-ng-db-free-mix-tests

fix/netflow-window-spec-map

fix/netflow-dashboard-covered-rate-denominator

fix/bgp-isolated-sample-markers

feat/srql-flow-interface-dimension

fix/netflow-dashboard-paired-interface-srql

fix/netflow-sankey-srql-other-rollup

fix/authored-flow-table-other-rollup

fix/netflow-topn-other-rollup

fix/authored-trend-epoch-sort

fix/interface-rate-series-rows

refactor/dashboard-sparkline-recent-wrapper

test/anomaly-capacity-timeout

fix/agent-addon-bazel-srcs

fix/agent-addon-stream-reconnect-jitter

fix/agent-addon-stable-stream-backoff-reset

fix/sysmon-cpu-core-max-chart

fix/sysmon-disk-mount-max-chart

fix/sysmon-native-chart-fanout-limit

fix/authored-query-time-window-clamp

test/web-ng-db-free-test-helper-cleanup

fix/anomaly-telemetry-drop-diagnostics

ci/gazelle-changed-go-drift-check

fix/sysmon-hottest-series-display-cap

fix/shared-srql-time-window-parsing

fix/authored-query-absolute-window-clamp

fix/anomaly-scoring-stall-health

fix/anomaly-counter-positive-rate-bound

fix/bgp-query-error-empty-state

fix/netflow-interface-rate-labels

test/srql-rate-counter-reset-gaps

test/seasonal-alert-open-clear-contract

test/web-ng-db-free-tag-filter

fix/anomaly-telemetry-drop-diagnostic-labels

ci/gazelle-drift-output-check

fix/sysmon-displayed-series-count-copy

fix/srql-time-window-single-quote-scrub

fix/authored-query-half-open-window-clamp

fix/anomaly-configurable-scoring-stale-threshold

test/anomaly-confirm-slot-boundaries

fix/anomaly-counter-nonpositive-elapsed

fix/bgp-generic-query-error-banner

fix/netflow-rate-label-resolved-window

docs/srql-rate-reset-wrap-semantics

test/seasonal-alert-contract-assertions

test/dashboard-live-harness-assertions

test/netflow-sankey-other-tail-semantics

test/causal-finding-upsert-fields

fix/srql-other-parsed-group-validation

fix/anomaly-capacity-query-timeout-opts

test/netflow-interface-p95-direction-max

fix/srql-flow-interface-fixture-e2e

docs/anomaly-subsystem-bug-hunt

fix-anomaly-engine-semantics-and-delivery

fix/advisory-feeds-demo-health

feat/cnpg-backup-chart

feat/cnpg-barman-backups

feat/vuln-intel-ui-rework

feat/wire-otel-collector-addon-build

fix/anomaly-addon-profile-schema

fix/seasonal-worker-dead-clause

fix/advisory-feeds-log-noise

release/v1.3.5

fix/cnpg-wal-cap-default

add-endpoint-inventory-addon-seeder

fix-web-ng-nats-object-store-acl

refactor/disposition-seasonal-module

feat/causal-disposition-capacity

refactor/gateway-metrics-publisher-base

feat/causal-disposition-seasonal

fix/addon-build-metric-feed-srcs

feat/snmp-agent-targeting

fix/snmp-counter-wire-type-gate

add-core-causal-disposition-nif

perf/anomaly-detector-fidelity

fix/anomaly-rekey-and-queries

perf/agent-commands-retention-index

fix/eventwriter-backpressure-hotpath

fix/web-ng-agent-artifact-url

fix/cnpg-image-digest

fix/anomaly-default-profile-query

fix/cnpg-default-digest

fix/native-addons-draft-release

fix/anomaly-addon-manifest-gate

feat/retire-central-anomaly

feat/edge-anomaly-addon

fix/core-elx-observability-runtime

fix/netprobe-disabled-ebpf-guard

fix/anomaly-engine-defaults

demo-prod-anomaly-engine

fix-anomaly-context-engine-sharded

demo-prod-nats-allowlist

fix-gateway-metrics-nats-allowlist

release/v1.3.4

feat-advisory-feeds-into-core

fix-dire-agent-link-reciprocal-anchor

fix-agent-version-ldflag-and-inventory-ownership

fix-demo-memory-limits

feat-addon-fleet-reporting

fix-capacity-persistence

fix-scalibr-stats-panic

feat-endpoint-packages-paginate-search

feat-endpoint-package-detail-modal

fix-agent-config-apply-cascade

deploy/eventwriter-fixes

fix-eventwriter-throughput-serialization

fix-endpoint-inventory-change-gated-deltas

fix-eventwriter-demand-flow-control

fix-addon-reconcile-agent-uid

deploy/staging-regression-fixes

fix/traces-discoverability

feat-process-table-paginate-search

fix-vuln-feed-default-urls

fix-addon-profile-reconcile

fix-endpoint-inventory-ingest

fix-endpoint-inventory-enablement

fix-addon-systemd-self-heal

fix-agent-config-reapply-storm

fix-jetstream-durable-recreate

fix-device-detail-ui

fix-capacity-forecast-math

fix-sysmon-process-metrics-visibility

release/v1.3.2-pre1

work/3796-review-fixes

work/3796-production-2m-shard-ready-batches

work/3788-rec10-reject-misbucketed-ocsf-source

work/3796-production-2m-compact-events

work/3796-production-2m-index-lookup

work/3788-remove-plugin-direct-metric-insert

work/3796-production-2m-shard-fanout

work/3788-rec1-plugin-metric-v2-envelope

work/3796-production-2m-batch-seen-inserts

work/3796-production-2m-defer-event-prune

work/3796-production-2m-benchmark-instrumentation

work/3796-anomaly-production-2m-spec

work/3796-deepcausality-impl

work/3788-remove-dead-metric-handlers

work/3788-rec10-consumer-metric-type-telemetry

work/v1.3.2-observability-metrics

work/3788-anomaly-scale-architecture

work/3788-compact-anomaly-evaluator

work/3789-3790-counter-cgroup-specs

work/3789-counter-normalizer

work/3796-deepcausality-reasoner

work/refactor-anomaly-reasoner-deepcausality

work/3789-snmp-raw-counters

work/cnpg-application-network-policy

work/anomaly-context-name-conflicts

work/3578-proxmox-host-identity

work/3578-retire-stale-agent-devices

work/3607-agent-command-json-payloads

work/3731-enable-causal-spine-demo

work/3742-security-finding-device-correlation

work/3733-capacity-forecast-source

work/3734-observability-health-view

work/3736-device-anomaly-capacity

work/3748-security-live-posture

work/3747-security-findings-payload

work/3727-3728-sysmon-metrics-fallback

work/causal-engine-msrv-1-90

work/3764-causalflow-anomaly-nif

work/causal-engine-deep-causality-graph

work/3713-causal-operator-rule-ttl

work/3741-cap-passive-netprobe-service-state-details

work/3742-security-finding-entity-rendering

work/3743-security-scanner-signal-freshness

work/3744-security-finding-identity-metadata

work/3745-preserve-dashboard-frame-results-on-error

work/3750-partial-sysmon-bulk-inserts

work/3751-reconcile-jetstream-consumers

work/3753-logical-gateway-id

work/3695-3696-connect-cache-zen-normalizer

work/3700-fence-anomaly-checkpoints

work/3737-3741-services-state-performance

work/3749-sysmon-legacy-service-check-bridge

work/3691-remove-dead-window-allocation

work/3693-rebuild-anomaly-context-from-insert

work/3694-coalesce-anomaly-checkpoints

work/3697-preserve-rehydrated-verdicts

work/3698-3699-capacity-retention-history

work/3701-3702-anomaly-config

work/3704-anomaly-severity-rollup

work/3705-3707-3708-3709-metrics-findings

work/3710-3711-3706-3703-causal-alerts

work/3712-preserve-causal-live-context

work/3713-causal-clearing-signals

renovate/alpine_3_20-3.20

work/3597-phase7.3-elixir-validation-cleanup

work/3597-phase7.2-live-otel-validation

work/3597-phase7.1-openspec-validation

work/3597-phase6.3-guarded-remediation-note

work/3597-phase6.2-anomaly-operator-docs

work/3597-phase6.1-jetstream-metrics-convention

work/3597-phase5.4-anomaly-ui-rbac

work/3597-phase5.3-netflow-placeholder-cleanup

work/3597-phase5.2-anomaly-findings-ui

work/3597-phase5.1-capacity-forecast-dashboard

work/3597-phase4.4-anomaly-settings-ui

work/3597-phase4.3-anomaly-config-hot-reload

work/3597-phase4.2-helm-seeded-anomaly-defaults

work/3597-phase4.1-anomaly-config-resources

work/3597-phase3.4-scaling-verification

work/3597-phase3.3-ocsf-finding-model-alignment

work/3597-phase3.2-causal-signal-alert-coverage

work/3597-phase3.1-causal-prediction-emission

work/3597-phase2.6-capacity-forecast-tests

work/3597-phase2.5-capacity-forecast-verdicts

work/3597-phase2.4-interface-capacity-denominator

work/3597-phase2.3-capacity-forecast-worker

work/3597-phase2.2-interface-hourly-rollup

work/3597-phase2.1-capacity-forecast-resource

work/3597-phase1.8-anomaly-detector-tests

work/3597-phase1.7-per-series-anomaly-config

work/3597-phase1.6-context-checkpoint-rehydration

work/3597-phase1.5-ingress-uuidv8-ordering

work/3597-phase1.4-anomaly-context-engine

work/3597-phase1.3-anomaly-analysis-consumer

work/3597-phase1.2-clean-baseline-reasoner

work/3597-phase1.1-causal-reasoner-nif

work/3597-phase0.12-required-otel-addon

work/3597-phase0.11-leaf-compatible-ingress

work/3597-phase0.10-sysmon-downsampling

work/3597-phase0.9-remove-self-loops

work/3597-phase0.8-defined-ingress

work/3597-phase0.7-metrics-cutover

work/3597-phase0.6-zen-nif

release/v1.3.1

fix/pin-time-0.3.46-async-nats-coherence

revert/time-0348-wrong-fix

fix/time-0348-rustc193

fix/flow-attribution-correlator-pressure

fix/trace-detail-param-limit

fix/security-dashboard-perf-and-falco-findings

proposal/add-causal-anomaly-detection

proposal/complete-security-analytics-pipeline

renovate/actions_runner

proposal/refactor-otel-signal-correlation

fix/proxmox-integration-connector-and-imports

demo/dire-rollout-ccacb3800

fix/dire-phases-2-8

openspec/refactor-device-identity-reconciliation

release/v1.2.99

fix/retry-transient-rekor-signing

fix/idempotent-cosign-tlog-conflict

release/v1.2.98

feat/pdns-protobuf-ocsf-addon

feat/add-agent-feature-sets-more

fix/netprobe-bazel-lock-0.2.21

chore/prepare-1.2.96-release

fix/netprobe-addon-version-bump

fix/netprobe-prioritize-active-metadata

feat/causal-engine-v1

release/1.2.95

fix/netprobe-event-driven-inventory

update-attributed-flow-explorer

release/1.2.94-netprobe-attribution

fix/nats-platform-object-store-acl

feat/attributed-flow-correlation

feat/netprobe-p0f-userspace-ebpf-verifier

add-endpoint-sbom-inventory

fix/remove-attributed-flow-fixtures

fix/demo-nats-import-acls

feat/netprobe-rollback-tests

fix/armis-northbound-raw-token-auth

fix/remove-stale-nginx

feat/native-addon-web-importer

feat/native-addon-artifact-mirror

feat/native-addon-importer-db-test

feat/native-addon-publish-pipeline

feat/netprobe-addon-carve

feat/addon-os-package-template

feat/addon-bundle-tarball

feat/addon-ephemeral-helper

feat/addon-pushed-tarball

feat/addon-systemd-dispatch

fix/elixir-quality-format

feat/addon-delivery-supervision

feat/native-addon-rust-sdk

feat/native-addon-build-signing

docs/reconcile-fingerprintd-netprobe-feature-sets

fix/addon-status-test-netprobe-fixture

feat/migrate-netprobe-native-addon

codex/fix-datasvc-nats-storage-acl

fix/rust-test-failures

fix/go-test-failures

feat/native-addon-edge-ops

feat/host-network-visibility-phase-2

feat/native-addon-delivery-models

feat/3425-agent-feature-sets-proposal

feat/3444-bumblebee-agent

fix/netprobe-e2e-sidecar-runtime

fix/srql-sort-diagnostics

feat/passive-device-fingerprinting

fix/device-list-sweep-availability

update/plugin-system

add-service-monitoring-foundation

fix/cli-auth-settings-navigation

fix-proxmox-console-react-client-render

fix-services-plugin-status-read-model

fix/docker-update/cnpg

fix/sweep-ip-family-routing

codex/remote-access-desktop-rdp

propose-interface-action-target-context

codex/fix-armis-names-string

fix/agent-accept-deprecated-remote-access-config

fix/device-results-count-facets

fix/bazelisk-installer-retries

fix/tinygo-host-toolchain-fetch

add-per-agent-availability

fix/forgejo-release-multipart-assets

fix/agent-config-stale-session

fix/mtr-hop-dns-resolution

fix/hostname-only-device-create

fix/otlp-log-metadata-sanitization

add-nats-object-store-retention

fix/helm-serviceradar-state-pvc

fix/armis-large-sync-streaming

demo/release-v1.2.44-source-fix

demo-rollout-proxmox-bazel-fix

security/postgres-update

fix/mtr-bulk-queue-and-srql-targets

armis-northbound-availability-updates

codex/topology-endpoint-evidence-investigation

codex/topology-bootstrap-and-layout-simplification

codex/remove-ingress-nginx-edge

security/k8s-hardening

2406-feat-agent-fleet-management-secure-self-update-system

chore/k8s-arc-update

rust-fix

2371-analytics-stats-cards-should-abbreviate-numbers

chore/perl-cleanup

192-feat-tftp-server

mikemiles-dev/feature/netflow_collection

815-feat-support-win32-for-agentpoller

gh-pages

v1.4.24

v1.4.23

v1.4.22

v1.4.21

v1.4.20

v1.4.19

v1.4.15

v1.4.14

v1.4.13

v1.4.12

v1.4.11

v1.4.10

v1.4.9

v1.4.8

v1.4.7

v1.4.6

sha-01c955ca30d49fdaa047b468396e76b9a0076cec

v1.4.5

sha-88b3cb84c59cc8b94317a7cdd944580fd951daf6

v1.4.4

sha-1723349b04c8f80a006a473497e5284f3a0bf86a

v1.4.3

sha-c4cc25916d1aa443360e8764883662b538bb9bc7

v1.4.2

v1.4.1

sha-888c4851ec54c13ff7e47648320bd0f404e0833b

sha-1deff86b2c5c73fb197958243dee68a68b6aec4b

v1.4.0

v1.3.10

v1.3.9

v1.3.8

v1.3.7

v1.3.6

v1.3.5

v1.3.5-pre1

v1.3.4

v1.3.3

v1.3.2

v1.3.2-pre1

v1.3.1

v1.3.0

v1.2.99

v1.2.98

v1.2.97

v1.2.96

v1.2.95

netprobe-0.2.20-demo

workload-identity-0.1.3-0649fc51a

netprobe-0.2.19-a89175a4e

sha-005a42346515d7e55c42c5d18018edc74f74bc6a

sha-983aceed91fa0768ce71503712ebac85776f937b

netprobe-0.2.11-288ed2f98

netprobe-0.2.10-100482679

netprobe-0.2.7-7aceecf23

netprobe-0.2.6-0592a4034

netprobe-0.2.5-e16b1e73a

netprobe-0.2.5-479b8dc71

netprobe-0.2.5-5f030f8d9

netprobe-0.2.3-458e203ae

netprobe-0.2.3-eab190932

v1.2.94

v1.2.93

v1.2.92

v1.2.91

sha-a808db901dd08e5f730ffe368af12eed9a5388af

v1.2.90

sha-2d851b470f300109a1e4ca6fda67a7886726c757

v1.2.89

v1.2.88

v1.2.87

v1.2.86

v1.2.85

sha-bc56da1a4b5f6233459e69def660d00df78a2fbc

v1.2.84

v1.2.83

v1.2.82

v1.2.81

v1.2.80

v1.2.79

v1.2.78

v1.2.77

v1.2.76

v1.2.75

v1.2.74

v1.2.73

v1.2.72

v1.2.71

v1.2.70

v1.2.69

v1.2.68

v1.2.67

v1.2.66

v1.2.65

v1.2.64

v1.2.63

v1.2.62

v1.2.61

v1.2.60

v1.2.59

v1.2.58

v1.2.57

v1.2.54

v1.2.53

v1.2.52

v1.2.51

v1.2.50

v1.2.49

v1.2.48

v1.2.47

v1.2.46

v1.2.45

v1.2.44

v1.2.43

v1.2.42

v1.2.41

v1.2.40

v1.2.39

v1.2.38

sha-de6d1025d59f039188754b895ff7fe65db9b306b

sha-8006b6105635acf43060fab2613eab3bccb1efcf

v1.2.37

v1.2.36

v1.2.35

v1.2.34

v1.2.33

v1.2.32

v1.2.31

v1.2.30

v1.2.29

v1.2.28

v1.2.27

v1.2.26

v1.2.25

v1.2.24

v1.2.23

v1.2.22

v1.2.21

v1.2.20

v1.2.19

v1.2.18

v1.2.17

v1.2.16

v1.2.15

v1.2.14

v1.2.13

v1.2.12

v1.2.11

v1.2.6

v1.2.10

v1.2.9

v1.2.8

v1.2.7

v1.2.5

v1.2.4

v1.2.3

v1.2.2

v1.2.1

v1.2.0

v1.1.2

v1.1.0

v1.0.92

v1.0.91

v1.0.90

v1.0.89

v1.0.88

v1.0.87

v1.0.86

v1.0.85

v1.0.84

v1.0.83

v1.0.82

v1.0.81

v1.0.78

v1.0.77

v1.0.76

v1.0.70

v1.0.69

v1.0.68

v1.0.67

v1.0.66

v1.0.65

v1.0.64

v1.0.63

v1.0.62

v1.0.61

v1.0.60

v1.0.59

v1.0.58

1.0.57

v1.0.56

v1.0.55

v1.0.54-pre5

v1.0.53

v1.0.53-pre19

v1.0.53-pre18

v1.0.53-pre17

v1.0.53-pre15

1.0.53-pre10

1.0.53-pre9

1.0.53-pre8

1.0.53-pre7

1.0.53-pre6

1.0.53-pre5

1.0.53-pre4

1.0.53-pre3

1.0.53-pre2

1.0.53-pre1

1.0.52

1.0.51

1.0.50

1.0.49

1.0.49-pre5

1.0.49-pre4

1.0.49-pre3

1.0.49-pre2

1.0.48

1.0.48-rc2

1.0.48-rc1

1.0.48-pre8

1.0.48-pre7

1.0.48-pre6

1.0.48-pre5

1.0.48-pre4

1.0.48-pre3

1.0.48-pre2

1.0.48-pre1

1.0.47

1.0.47-pre8

1.0.47-pre7

1.0.47-pre6

1.0.47-pre5

1.0.47-pre4

1.0.47-pre3

1.0.47-pre2

1.0.47-pre1

1.0.46

1.0.46-pre9

1.0.46-pre8

1.0.46-pre7

1.0.46-pre6

1.0.46-pre5

1.0.46-pre4

1.0.46-pre3

1.0.46-pre2

1.0.46-pre1

1.0.45

1.0.44

1.0.44-pre12

1.0.44-pre11

1.0.44-pre10

1.0.44-pre9

1.0.44-pre8

1.0.44-pre7

1.0.44-pre6

1.0.44-pre5

1.0.44-pre4

1.0.44-pre3

1.0.44-pre2

1.0.44-pre1

1.0.43

1.0.42

1.0.41

1.0.41-pre1

1.0.40

1.0.40-pre11

1.0.40-pre10

1.0.40-pre9

1.0.40-pre8

1.0.40-pre7

1.0.40-pre6

1.0.40-pre5

1.0.40-pre4

1.0.40-pre3

1.0.40-pre2

1.0.40-pre1

1.0.39

1.0.38

1.0.37

1.0.36

1.0.36-pre5

1.0.36-pre4

1.0.36-pre3

1.0.36-pre2

1.0.35

1.0.35-pre3

1.0.35-pre2

1.0.35-pre

1.0.34-pre3

1.0.34-pre2

1.0.34-pre1

1.0.33

1.0.33-pre2

1.0.33-pre

1.0.32

1.0.31

1.0.30

1.0.29

1.0.28

1.0.27

1.0.26

1.0.25

1.0.24

1.0.23

1.0.22

1.0.21

1.0.20

1.0.19

1.0.18

1.0.17

1.0.16

1.0.15

1.0.14

1.0.13

1.0.11

1.0.10

1.0.9

1.0.8

1.0.7

1.0.6

1.0.5

1.0.4

1.0.3

1.0.2

1.0.1

1.0.0

Labels

Clear labels   

1week

  

2weeks

  

Failed compliance check

  

IP cameras

  

NATS

NATS JetStream

  

Possible security concern

  

Review effort 1/5

  

Review effort 2/5

  

Review effort 3/5

  

Review effort 4/5

  

Review effort 5/5

  

UI

  

aardvark

  

accessibility

  

amd64

  

api

  

arm64

  

auth

  

back-end

  

bgp

  

blog

  

bug

Something isn't working

  

build

  

checkers

  

ci-cd

continuous integration-continuous deployments

  

cleanup

  

cnpg

cloud-native postgres

  

codex

  

core

core service

  

dependencies

Pull requests that update a dependency file

  

device-management

  

documentation

Improvements or additions to documentation

  

duplicate

This issue or pull request already exists

  

dusk

  

ebpf

  

enhancement

New feature or request

  

eta 1d

  

eta 1hr

  

eta 3d

  

eta 3hr

  

feature

  

fieldsurvey

  

github_actions

Pull requests that update GitHub Actions code

  

go

Pull requests that update Go code

  

good first issue

Good for newcomers

  

help wanted

Extra attention is needed

  

invalid

This doesn't seem right

  

javascript

Pull requests that update Javascript code

  

k8s

  

log-collector

  

mapper

  

mtr

multi traceroute

  

needs-triage

  

netflow

  

network-sweep

  

observability

  

oracle

Oracle Linux related issues

  

otel

opentelemetry logs, traces, metrics

  

plug-in

  

proton

timeplus proton streaming database

  

python

  

question

Further information is requested

  

reddit

  

redhat

  

research

  

rperf

  

rperf-checker

  

rust

Pull requests that update rust code

  

sdk

  

security

  

serviceradar-agent

  

serviceradar-agent-gateway

  

serviceradar-web

  

serviceradar-web-ng

  

siem

  

snmp

  

sysmon

  

topology

  

ubiquiti

  

wasm

  

wontfix

This will not be worked on

  

zen-engine

No labels

1week

2weeks

Failed compliance check

IP cameras

NATS

Possible security concern

Review effort 1/5

Review effort 2/5

Review effort 3/5

Review effort 4/5

Review effort 5/5

UI

aardvark

accessibility

amd64

api

arm64

auth

back-end

bgp

blog

bug

build

checkers

ci-cd

cleanup

cnpg

codex

core

dependencies

device-management

documentation

duplicate

dusk

ebpf

enhancement

eta 1d

eta 1hr

eta 3d

eta 3hr

feature

fieldsurvey

github_actions

go

good first issue

help wanted

invalid

javascript

k8s

log-collector

mapper

mtr

needs-triage

netflow

network-sweep

observability

oracle

otel

plug-in

proton

python

question

reddit

redhat

research

rperf

rperf-checker

rust

sdk

security

serviceradar-agent

serviceradar-agent-gateway

serviceradar-web

serviceradar-web-ng

siem

snmp

sysmon

topology

ubiquiti

wasm

wontfix

zen-engine

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

carverauto/serviceradar#3788

No description provided.