P1: OCSF 1.9 finding model — separate Findings from Events across security sources #3594

New issue

Open

opened 2026-06-12 02:00:03 +00:00 by mfreeman451 · 0 comments

mfreeman451 commented

2026-06-12 02:00:03 +00:00

Owner

Handoff: OCSF 1.9 finding model — separate Findings from Events

Priority: P1 — architecture correction for security analytics.
Type: OpenSpec proposal + implementation (Elixir core schema/Ash + migrations + SRQL + web-ng).

This is the proposal the OCSF implementation agent should turn into an OpenSpec
change (suggest change-id add-ocsf-finding-model) and implement. It supersedes
/ deepens the finding portions of the already-approved
complete-security-analytics-pipeline proposal (PR #3589); coordinate so they
don't conflict (that proposal's dashboard-perf + falco-promotion quick-wins
already shipped in PR #3591).

Problem (user-stated)

ServiceRadar claims to turn Falco (and other security signals) into "OCSF
findings", but it really just writes events — every security signal is an
ocsf_events row, and Falco is shoehorned into that table tagged
class_uid=2004 (detection_finding) with all the finding-specific data buried in
a JSONB metadata.security_signal.diagnostics blob. There is no Finding
entity. The correct OCSF model is: a Finding is a stateful, deduplicated,
correlated object (one logical finding, updated over time) and Events
reference the finding (finding_info.uid), rather than each occurrence being a
fresh pseudo-finding row in the events table.

This wrong shape must be fixed for Falco AND audited/fixed across all security
sources (trivy, bumblebee, vulnerability advisory feeds, scan_activity,
dns_activity).

Authoritative OCSF 1.9 finding classes to model

Author faithful schemas for these (from https://schema.ocsf.io/1.9.0-dev/classes/...):

vulnerability_finding (class_uid 2002) — CVE/CVSS, vulnerabilities[],
affected_packages/CPE/PURL, remediation. → trivy, vulnerability advisory feeds, endpoint-inventory matches.
compliance_finding (2003) — compliance/checks, control, status. → policy/posture checks.
detection_finding (2004) — detections, evidence, attacks[] (MITRE
tactic/technique), kill_chain. → falco runtime detections.
incident_finding (2005) — incident grouping/state. → correlated incidents (stateful alerts).
data_security_finding (2006) — DLP/data classification. → future.
application_security_posture_finding (2007) — ASPM/SAST/DAST posture. → future.

Each shares the OCSF base Finding structure: finding_info (uid, title,
types, src_url, related_events, first_seen/last_seen), activity_id
(Create/Update/Close), status/status_id, severity_id, confidence_id,
risk_score, remediation, evidence. The implementer MUST fetch the live
1.9.0-dev schema pages (WebFetch on the class URL; JSON at
https://schema.ocsf.io/api/1.9.0-dev/classes/<name> or the ?extensions=
variant; sample at /sample/1.9.0-dev/classes/<name>) to get the exact required
attributes and nested objects — do not guess field lists.

Required architecture

A Finding entity distinct from Events. New CNPG table(s) + Ash
resource(s) for findings keyed by a stable finding_info.uid (deterministic
from class + rule/check/CVE + key entity dimensions so re-occurrences UPDATE
one finding: first_seen/last_seen, occurrence count, current status). Decide:
one polymorphic ocsf_findings table with class-specific JSONB +
first-class common columns, vs per-class tables. Recommend a single
ocsf_findings hypertable/table with first-class columns for the common
Finding base (uid, class_uid, severity_id, status_id, risk_score, title,
first_seen, last_seen, finding_info, entity refs) and a class-typed JSONB for
class-specific objects (vulnerabilities/compliance/evidence) — queryable +
extensible. Justify in design.md.
Events reference findings. An ocsf_events row for a security signal
carries finding_info.uid (FK-ish reference) instead of being the finding.
The event = "this occurred at time T"; the finding = the durable, deduped
object. Promotion creates/updates the finding, then writes an event referencing it.
Per-source mapping onto the correct class (the audit deliverable —
confirm/refine against code):
- falco → detection_finding (2004), MITRE tags → attacks[].
- trivy → vulnerability_finding (2002).
- vulnerability advisory feeds (NVD/CISA-KEV/VulnCheck) + endpoint-inventory
  package matches → vulnerability_finding (2002).
- bumblebee → detection_finding or vulnerability_finding (confirm what it produces).
- stateful-alert incidents → incident_finding (2005), referencing constituent findings.
SRQL + UI: in:security_findings should query the FINDINGS entity (deduped,
stateful), with a way to drill into the referencing events. Keep
scan_activity/dns_activity as activity-event entities. Update the SRQL
catalog + the security dashboard frames to read findings vs events correctly.
MITRE ATT&CK: parse Falco rule tags (e.g. T1059, mitre_execution)
into structured attacks[] (tactic/technique) on the detection_finding —
currently captured raw but never structured.

Current-state evidence (from audit)

ocsf_events migration: generic columns only (severity_id, message, metadata
JSONB, observables JSONB, actor/device/src_endpoint/dst_endpoint JSONB) — no
finding columns.
elixir/serviceradar_core/priv/repo/migrations/20260203120000_create_ocsf_events.exs:12-46.
Falco decomposition (rich, but into the JSONB blob, class hardcoded 2004):
elixir/serviceradar_core/lib/serviceradar/event_writer/processors/falco_events.ex
(diagnostics build ~429-616; class ~285-289; event UUID ~754-766 gives no
finding dedup).
Duplicate falco decomposition path in log_promotion.ex:313-616 — extract a
shared FalcoDecomposition module.
OCSF helper lacking finding builders: event_writer/ocsf.ex (no
finding_info/remediation/risk_score/confidence/kill_chain helpers).
SRQL entities security_findings/scan_activity/dns_activity in the catalog.

Migration plan requirements

Backfill/transition existing falco/trivy/etc. ocsf_events into the new findings
model without breaking the security dashboard or /security. New writes go
through the finding-create/update path; consider a one-time backfill that
derives findings from existing finding-shaped events (or accept forward-only +
retention ages out the old shape — decide and document).
BREAKING: in:security_findings result shape changes (finding columns); SRQL
catalog + dashboard frames + any consumers updated in lockstep.

Build note

Same as other Elixir work: avoid cold mix compile stalls in long agents; format

targeted compiles; CI compiles. Ash codegen for migrations per the repo's
mix ash.codegen workflow (NOT mix ecto.gen.migration); special tables
(hypertables) use raw SQL migration + migrate? false Ash resource per
elixir/serviceradar_core/CLAUDE.md.

Done-when

6 OCSF 1.9 finding classes modeled faithfully (fields verified against schema.ocsf.io).
A Finding entity exists, stateful + deduped; Events reference findings.
Falco/trivy/feeds/bumblebee map to the correct class; MITRE structured.
SRQL + security dashboard read findings vs events correctly.
OpenSpec change validates --strict.

# Handoff: OCSF 1.9 finding model — separate Findings from Events **Priority: P1 — architecture correction for security analytics.** **Type: OpenSpec proposal + implementation (Elixir core schema/Ash + migrations + SRQL + web-ng).** This is the proposal the OCSF implementation agent should turn into an OpenSpec change (suggest change-id `add-ocsf-finding-model`) and implement. It supersedes / deepens the finding portions of the already-approved `complete-security-analytics-pipeline` proposal (PR #3589); coordinate so they don't conflict (that proposal's dashboard-perf + falco-promotion quick-wins already shipped in PR #3591). ## Problem (user-stated) ServiceRadar claims to turn Falco (and other security signals) into "OCSF findings", but it really just writes **events** — every security signal is an `ocsf_events` row, and Falco is shoehorned into that table tagged `class_uid=2004` (detection_finding) with all the finding-specific data buried in a JSONB `metadata.security_signal.diagnostics` blob. There is **no Finding entity**. The correct OCSF model is: a **Finding** is a stateful, deduplicated, correlated object (one logical finding, updated over time) and **Events** *reference* the finding (`finding_info.uid`), rather than each occurrence being a fresh pseudo-finding row in the events table. This wrong shape must be fixed for Falco AND audited/fixed across all security sources (trivy, bumblebee, vulnerability advisory feeds, scan_activity, dns_activity). ## Authoritative OCSF 1.9 finding classes to model Author faithful schemas for these (from https://schema.ocsf.io/1.9.0-dev/classes/...): - `vulnerability_finding` (class_uid 2002) — CVE/CVSS, `vulnerabilities[]`, affected_packages/CPE/PURL, remediation. → trivy, vulnerability advisory feeds, endpoint-inventory matches. - `compliance_finding` (2003) — compliance/checks, control, status. → policy/posture checks. - `detection_finding` (2004) — detections, `evidence`, `attacks[]` (MITRE tactic/technique), kill_chain. → falco runtime detections. - `incident_finding` (2005) — incident grouping/state. → correlated incidents (stateful alerts). - `data_security_finding` (2006) — DLP/data classification. → future. - `application_security_posture_finding` (2007) — ASPM/SAST/DAST posture. → future. Each shares the OCSF **base Finding** structure: `finding_info` (uid, title, types, src_url, related_events, first_seen/last_seen), `activity_id` (Create/Update/Close), `status`/`status_id`, `severity_id`, `confidence_id`, `risk_score`, `remediation`, `evidence`. The implementer MUST fetch the live 1.9.0-dev schema pages (WebFetch on the class URL; JSON at `https://schema.ocsf.io/api/1.9.0-dev/classes/<name>` or the `?extensions=` variant; sample at `/sample/1.9.0-dev/classes/<name>`) to get the exact required attributes and nested objects — do not guess field lists. ## Required architecture 1. **A Finding entity distinct from Events.** New CNPG table(s) + Ash resource(s) for findings keyed by a stable `finding_info.uid` (deterministic from class + rule/check/CVE + key entity dimensions so re-occurrences UPDATE one finding: first_seen/last_seen, occurrence count, current status). Decide: one polymorphic `ocsf_findings` table with class-specific JSONB + first-class common columns, vs per-class tables. Recommend a single `ocsf_findings` hypertable/table with first-class columns for the common Finding base (uid, class_uid, severity_id, status_id, risk_score, title, first_seen, last_seen, finding_info, entity refs) and a class-typed JSONB for class-specific objects (vulnerabilities/compliance/evidence) — queryable + extensible. Justify in design.md. 2. **Events reference findings.** An `ocsf_events` row for a security signal carries `finding_info.uid` (FK-ish reference) instead of being the finding. The event = "this occurred at time T"; the finding = the durable, deduped object. Promotion creates/updates the finding, then writes an event referencing it. 3. **Per-source mapping onto the correct class** (the audit deliverable — confirm/refine against code): - falco → detection_finding (2004), MITRE tags → `attacks[]`. - trivy → vulnerability_finding (2002). - vulnerability advisory feeds (NVD/CISA-KEV/VulnCheck) + endpoint-inventory package matches → vulnerability_finding (2002). - bumblebee → detection_finding or vulnerability_finding (confirm what it produces). - stateful-alert incidents → incident_finding (2005), referencing constituent findings. 4. **SRQL + UI**: `in:security_findings` should query the FINDINGS entity (deduped, stateful), with a way to drill into the referencing events. Keep `scan_activity`/`dns_activity` as activity-event entities. Update the SRQL catalog + the security dashboard frames to read findings vs events correctly. 5. **MITRE ATT&CK**: parse Falco rule `tags` (e.g. `T1059`, `mitre_execution`) into structured `attacks[]` (tactic/technique) on the detection_finding — currently captured raw but never structured. ## Current-state evidence (from audit) - `ocsf_events` migration: generic columns only (severity_id, message, metadata JSONB, observables JSONB, actor/device/src_endpoint/dst_endpoint JSONB) — no finding columns. `elixir/serviceradar_core/priv/repo/migrations/20260203120000_create_ocsf_events.exs:12-46`. - Falco decomposition (rich, but into the JSONB blob, class hardcoded 2004): `elixir/serviceradar_core/lib/serviceradar/event_writer/processors/falco_events.ex` (diagnostics build ~429-616; class ~285-289; event UUID ~754-766 gives no finding dedup). - Duplicate falco decomposition path in `log_promotion.ex:313-616` — extract a shared `FalcoDecomposition` module. - OCSF helper lacking finding builders: `event_writer/ocsf.ex` (no finding_info/remediation/risk_score/confidence/kill_chain helpers). - SRQL entities security_findings/scan_activity/dns_activity in the catalog. ## Migration plan requirements - Backfill/transition existing falco/trivy/etc. ocsf_events into the new findings model without breaking the security dashboard or `/security`. New writes go through the finding-create/update path; consider a one-time backfill that derives findings from existing finding-shaped events (or accept forward-only + retention ages out the old shape — decide and document). - BREAKING: `in:security_findings` result shape changes (finding columns); SRQL catalog + dashboard frames + any consumers updated in lockstep. ## Build note Same as other Elixir work: avoid cold `mix compile` stalls in long agents; format + targeted compiles; CI compiles. Ash codegen for migrations per the repo's `mix ash.codegen` workflow (NOT `mix ecto.gen.migration`); special tables (hypertables) use raw SQL migration + `migrate? false` Ash resource per elixir/serviceradar_core/CLAUDE.md. ## Done-when - 6 OCSF 1.9 finding classes modeled faithfully (fields verified against schema.ocsf.io). - A Finding entity exists, stateful + deduped; Events reference findings. - Falco/trivy/feeds/bumblebee map to the correct class; MITRE structured. - SRQL + security dashboard read findings vs events correctly. - OpenSpec change validates `--strict`.

mfreeman451 referenced this issue

2026-06-12 05:00:15 +00:00

Fix native add-on blob retention and move telemetry onboarding #3600