Anomaly ingest: derive p50/p95 from OTLP histogram points (stop dropping value=nil) [#3788 REC6b] #3800

Open
opened 2026-06-13 23:30:17 +00:00 by mfreeman451 · 0 comments
Owner

Anomaly ingest: derive p50/p95 from OTLP histogram points (stop dropping value=nil)

Part of carverauto/serviceradar#3788 REC6b. Not covered by any existing task (#3789 counters / #3790 cgroup / #3791 cadence specs only mention histograms at the contract level).

Today OTLP histogram data points (count/sum/bucket_counts/explicit_bounds) are emitted with value: nil and then dropped at anomaly extraction:

  • event_writer/processors/otel_metrics.ex:289-298,321-325 — histogram rows carry value: nil.
  • anomaly_detection/sample_extractor.exotel_sample only matches value/duration_ms (:208,:220); histogram points fall to the catch-all and return [] (:232).

So latency/size distributions never produce an anomaly series.

Ask

  • In sample_extractor, ingest histogram count/sum/bucket_counts/explicit_bounds and derive percentile series (p50/p95) for anomaly evaluation instead of rejecting value=nil.
  • Decide storage: extend timeseries_metrics with histogram columns, or force histograms to otel_metric_points (which already stores bucket_counts/explicit_bounds) — see #3788 open questions.
  • Exponential histograms / summaries are also counted-then-dropped in otel_metrics.ex — scope whether to handle here or defer.
## Anomaly ingest: derive p50/p95 from OTLP histogram points (stop dropping value=nil) Part of carverauto/serviceradar#3788 REC6b. **Not covered by any existing task** (#3789 counters / #3790 cgroup / #3791 cadence specs only mention histograms at the contract level). Today OTLP **histogram** data points (`count`/`sum`/`bucket_counts`/`explicit_bounds`) are emitted with `value: nil` and then dropped at anomaly extraction: - `event_writer/processors/otel_metrics.ex:289-298,321-325` — histogram rows carry `value: nil`. - `anomaly_detection/sample_extractor.ex` — `otel_sample` only matches `value`/`duration_ms` (~`:208`,`:220`); histogram points fall to the catch-all and return `[]` (~`:232`). So latency/size distributions never produce an anomaly series. ### Ask - In `sample_extractor`, ingest histogram `count`/`sum`/`bucket_counts`/`explicit_bounds` and derive percentile series (p50/p95) for anomaly evaluation instead of rejecting `value=nil`. - Decide storage: extend `timeseries_metrics` with histogram columns, or force histograms to `otel_metric_points` (which already stores `bucket_counts`/`explicit_bounds`) — see #3788 open questions. - Exponential histograms / summaries are also counted-then-dropped in `otel_metrics.ex` — scope whether to handle here or defer.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
carverauto/serviceradar#3800
No description provided.