Session status: open PRs + demo rebuild/release procedure + handoff index #3596

New issue

Open

opened 2026-06-12 02:00:03 +00:00 by mfreeman451 · 0 comments

mfreeman451 commented

2026-06-12 02:00:03 +00:00

Owner

Handoff: session status, open PRs, and the demo rebuild / release

This session did a large OTel correlation overhaul + a wave of demo bug fixes.
Below is what's landed, what's open, the remaining implementation handoffs, and
the rebuild/release procedure. Implementation is being handed to another agent.

Open PRs from this session (verify/merge, then rebuild)

#3592 Fix agent rollout UI: show failed targets, stop stale Pending — web-ng
#3591 Speed up dashboards (concurrent frames) + surface low-severity Falco findings — web-ng + core (approved security quick-wins)
#3590 Don't let failed add-on artifact delivery block agent config ack — go agent (poison-pill; lint fixed)
#3589 Proposal: complete the security analytics pipeline — OpenSpec (APPROVED)
#3588 Enable Go service trace export in demo for cross-component traces — go + helm
branch fix/traces-discoverability (pushed; PR may need creating) — spans column + sort + multi-span chip + trace-detail SRQL bar
Earlier merged this session: #3581 (release build fix), #3582 (precommit fmt), #3583 (skill doc), #3584 (sweep ingestor), #3585 (one-time-reset migration), #3586 (clickable trace/span IDs). Confirm merge state.

Remaining implementation handoffs (separate .md files in this dir)

01-addon-blob-eviction.md — P0, keystone. The native add-on distribution
system silently evicts in-use artifact blobs (JetStream serviceradar-objects
2 GiB + DiscardOld + no native-addons/* retention). Fixes: DiscardNew +
bigger bucket + reference-aware retention + blob-liveness surfacing.
02-ocsf-finding-model.md — P1. Separate OCSF Findings from Events; author
the 6 OCSF 1.9 finding classes; map all security sources to the right class;
MITRE structuring. (OpenSpec change add-ocsf-finding-model; deepens #3589.)
03-telemetry-onboarding-under-edge-ops.md — P3. Move the telemetry
onboarding LiveView under the edge-ops settings section.

Live demo findings (context, mostly addressed)

Agent binary rollout was actually COMPLETE (DB healthy); the UI "stuck Pending"
- "invisible failed target" were the two web-ng bugs fixed in #3592, downstream
  of the add-on poison-pill (#3590) + the blob eviction (handoff #01).
Cross-component traces: only core-elx + web-ng emit spans today; Go services
(agent/datasvc/db-event-writer) have OTel tracing DISABLED in the deployed
config. #3588 enables it. After deploy, otelgrpc auto-links agent→gateway→core.
(web-ng is a separate read-path trace root — won't join the agent push chain.)
Endpoint inventory not running = the blob 404 (handoff #01) + scalibr add-on
postdates v1.2.99 so it's in no release. Activation = release + UI-import +
Endpoint Inventory profile assign. A v1.3.0 release + the #01 fix is required.
Device-identity reconciliation atom crash: already fixed in staging (commit
f21424565), ships with the rebuild.
Vulnerability Intelligence empty = the NVD/CISA-KEV/VulnCheck producer packages
were never authored (contract/ingestor/UI all exist). Covered by proposal #3589.

Demo rebuild + release procedure (when PRs are merged)

The proper path is a real release (it publishes native add-on artifacts to OCI +
the import index, which — with handoff #01 — fixes the add-on distribution):

Merge the open PRs above into staging; ensure staging CI is green.
On staging: update CHANGELOG (a v1.3.0 entry is drafted in the otel worktree's
working tree — port it), bump VERSION to 1.3.0, run
scripts/cut-release.sh --version 1.3.0 --push.
The v1.3.0 tag triggers .forgejo/workflows/native-addons.yml (builds/signs/
pushes add-on tarballs + import index) and argocd-image-updater auto-deploys
demo on the v* tag (returns demo from the test sha-tag to the semver path).
Re-import native add-ons from the v1.3.0 release in web-ng so AddonPackages are
approved with live blobs (handoff #01 keeps them from being evicted).
Assign endpoint-inventory via Settings → Agents → Endpoint Inventory profile.
Validate: scripts/otel-conformance.sh against the OTLP gateway endpoint;
confirm agent→gateway→core multi-service traces; endpoint inventory populates;
security dashboard loads fast; falco low-severity findings appear.

Demo rollout note (sha-tag path used this session)

Demo was rolled to sha-5fbe93965… via the demo-local-rollout skill (host-crane
push workaround for macOS; OpenBao cosign; demo/prod-release branch advance +
argocd targetRevision pin). The proper v1.3.0 release supersedes that.

Ops follow-ups (not code)

gitops PR #34 (external-dns gateway-tlsroute) — merge + apply so the OTLP gRPC
gateway hostname (otlp-demo.grpc.serviceradar.cloud) gets DNS.
demo-staging namespace needs label serviceradar.com/gateway-access=true for
its OTLP gateway routes to attach.
Upstream opentelemetry-erlang PR #1001 (the double-hex log-id fix) — EasyCLA.

Build gotcha for implementers

Cold mix compile (rebuilds the SRQL Rust NIF) takes minutes and stalls
long-running agents on the 600s watchdog. Do Elixir work with mix format +
careful editing and rely on CI to compile, OR work in a warm worktree. Go/Rust
compiles are fine. New Go/Elixir/Rust files need BUILD.bazel updates (bazel test
breaks even when go test/mix test pass).

# Handoff: session status, open PRs, and the demo rebuild / release This session did a large OTel correlation overhaul + a wave of demo bug fixes. Below is what's landed, what's open, the remaining implementation handoffs, and the rebuild/release procedure. Implementation is being handed to another agent. ## Open PRs from this session (verify/merge, then rebuild) - **#3592** Fix agent rollout UI: show failed targets, stop stale Pending — web-ng - **#3591** Speed up dashboards (concurrent frames) + surface low-severity Falco findings — web-ng + core (approved security quick-wins) - **#3590** Don't let failed add-on artifact delivery block agent config ack — go agent (poison-pill; lint fixed) - **#3589** Proposal: complete the security analytics pipeline — OpenSpec (APPROVED) - **#3588** Enable Go service trace export in demo for cross-component traces — go + helm - branch `fix/traces-discoverability` (pushed; PR may need creating) — spans column + sort + multi-span chip + trace-detail SRQL bar - Earlier merged this session: #3581 (release build fix), #3582 (precommit fmt), #3583 (skill doc), #3584 (sweep ingestor), #3585 (one-time-reset migration), #3586 (clickable trace/span IDs). Confirm merge state. ## Remaining implementation handoffs (separate .md files in this dir) 1. `01-addon-blob-eviction.md` — **P0**, keystone. The native add-on distribution system silently evicts in-use artifact blobs (JetStream `serviceradar-objects` 2 GiB + DiscardOld + no `native-addons/*` retention). Fixes: DiscardNew + bigger bucket + reference-aware retention + blob-liveness surfacing. 2. `02-ocsf-finding-model.md` — **P1**. Separate OCSF Findings from Events; author the 6 OCSF 1.9 finding classes; map all security sources to the right class; MITRE structuring. (OpenSpec change `add-ocsf-finding-model`; deepens #3589.) 3. `03-telemetry-onboarding-under-edge-ops.md` — **P3**. Move the telemetry onboarding LiveView under the edge-ops settings section. ## Live demo findings (context, mostly addressed) - Agent binary rollout was actually COMPLETE (DB healthy); the UI "stuck Pending" + "invisible failed target" were the two web-ng bugs fixed in #3592, downstream of the add-on poison-pill (#3590) + the blob eviction (handoff #01). - Cross-component traces: only core-elx + web-ng emit spans today; Go services (agent/datasvc/db-event-writer) have OTel tracing DISABLED in the deployed config. #3588 enables it. After deploy, otelgrpc auto-links agent→gateway→core. (web-ng is a separate read-path trace root — won't join the agent push chain.) - Endpoint inventory not running = the blob 404 (handoff #01) + scalibr add-on postdates v1.2.99 so it's in no release. Activation = release + UI-import + Endpoint Inventory profile assign. A v1.3.0 release + the #01 fix is required. - Device-identity reconciliation atom crash: already fixed in staging (commit f21424565), ships with the rebuild. - Vulnerability Intelligence empty = the NVD/CISA-KEV/VulnCheck producer packages were never authored (contract/ingestor/UI all exist). Covered by proposal #3589. ## Demo rebuild + release procedure (when PRs are merged) The proper path is a real release (it publishes native add-on artifacts to OCI + the import index, which — with handoff #01 — fixes the add-on distribution): 1. Merge the open PRs above into staging; ensure staging CI is green. 2. On staging: update CHANGELOG (a v1.3.0 entry is drafted in the otel worktree's working tree — port it), bump VERSION to 1.3.0, run `scripts/cut-release.sh --version 1.3.0 --push`. 3. The `v1.3.0` tag triggers `.forgejo/workflows/native-addons.yml` (builds/signs/ pushes add-on tarballs + import index) and argocd-image-updater auto-deploys demo on the `v*` tag (returns demo from the test sha-tag to the semver path). 4. Re-import native add-ons from the v1.3.0 release in web-ng so AddonPackages are approved with live blobs (handoff #01 keeps them from being evicted). 5. Assign endpoint-inventory via Settings → Agents → Endpoint Inventory profile. 6. Validate: `scripts/otel-conformance.sh` against the OTLP gateway endpoint; confirm agent→gateway→core multi-service traces; endpoint inventory populates; security dashboard loads fast; falco low-severity findings appear. ## Demo rollout note (sha-tag path used this session) Demo was rolled to `sha-5fbe93965…` via the demo-local-rollout skill (host-crane push workaround for macOS; OpenBao cosign; demo/prod-release branch advance + argocd targetRevision pin). The proper v1.3.0 release supersedes that. ## Ops follow-ups (not code) - gitops PR #34 (external-dns gateway-tlsroute) — merge + apply so the OTLP gRPC gateway hostname (`otlp-demo.grpc.serviceradar.cloud`) gets DNS. - `demo-staging` namespace needs label `serviceradar.com/gateway-access=true` for its OTLP gateway routes to attach. - Upstream opentelemetry-erlang PR #1001 (the double-hex log-id fix) — EasyCLA. ## Build gotcha for implementers Cold `mix compile` (rebuilds the SRQL Rust NIF) takes minutes and stalls long-running agents on the 600s watchdog. Do Elixir work with `mix format` + careful editing and rely on CI to compile, OR work in a warm worktree. Go/Rust compiles are fine. New Go/Elixir/Rust files need BUILD.bazel updates (bazel test breaks even when `go test`/`mix test` pass).

mfreeman451 referenced this issue

2026-06-12 05:00:15 +00:00

Fix native add-on blob retention and move telemetry onboarding #3600