carverauto/serviceradar

Fork 0

DIRE 48h post-merge checkup — verify blob oscillation + apply agent-links remediation #3578

New issue

Open

opened 2026-06-11 04:27:34 +00:00 by mfreeman451 · 5 comments

mfreeman451 commented

2026-06-11 04:27:34 +00:00

Owner

Context

48-hour follow-up after merging the DIRE proxmox-connector + identity-model fix series (merged 2026-06-11). Verify the merged fixes held under live traffic and apply the remaining (dry-run-validated) remediation.

OpenSpec proposal: openspec/changes/refactor-device-identity-reconciliation/
Memory: project_dire_proxmox_identity, feedback_network_agnostic_identity.

Merged PRs (all in `staging`, code.carverauto.dev)

#3573 — reconcile_duplicates atom-crash fix + :maintenance off web-ng
#3574 — skip IP-less proxmox guests + honor rule scope_value in credential test
#3575 — proxmox guest IP/MAC by configured NIC (not ClusterIP VIP) + converge drifted plugin assignments (connector revival)
#3577 — agent↔device anchor on agent_id + chimera guard + behavioral IP-merge guard (distinct-MAC, network-agnostic)

Verify at +48h (target 2026-06-13)

Blob oscillation bounded: SELECT count(*) FROM platform.device_identifiers; should stay ~150K (was 12.3M→149K), not climb.
No new chimeras: connected agents on a device with >1 distinct armis_device_id or >32 MACs should be ~0.
Connector alive: proxmox virtualization card refreshing; assignment reconcile not crashing.

Remaining remediation to APPLY (validated dry-run only)

Validated live via .agents/skills/demo-cnpg-local-web-ng (DB route 10.0.2.12:32040; creds demo/serviceradar-db-credentials). agent-links dry-run plans:

k8s-agent: create clean device (off the 60-MAC FAKER-HP-WOR-25900 chimera sr:65b08097)
agent-k8s-cp2-worker3: restore off sr:6a62a996

Apply with: DireRemediation.run(mode: :execute, steps: ["agent-links"]) (writes a rollback NDJSON manifest). Re-run dry-run first to re-confirm.

Open follow-ups (NOT yet fixed — new work)

tonka01 bolt: ~12 phantom "dusk" agents from agent-UID churn (agent-dusk → agent-agent-dusk [double-prefix bug] → agent-dusk01 + transient -NNNN suffixes), each pinned to a stray device; agent-dusk bolts tonka01 (sr:129bcb1f). Needs a "exactly one agent record per agent — reap stale duplicates" remediation step (dry-run-first). Find the double-prefix source (the inferred_agent_id guards in falco/trivy are idempotent, so it's elsewhere).
pve host duplicates: each pve node has one device with integration_id=proxmox:hypervisor:pveNN but no IP, plus N bare-IP devices (no node id) from network discovery → disjoint namespaces, never merge. Fix: proxmox node enrichment must emit node IP/MAC (host-level version of #3575) so the node device merges with the IP-discovered records.

How to validate

.agents/skills/demo-cnpg-local-web-ng/scripts/start-local-web-ng.sh, or one-shot mix run -e '<code>' in elixir/web-ng with CNPG_HOST=10.0.2.12 CNPG_PORT=32040 CNPG_DATABASE=serviceradar CNPG_USERNAME=serviceradar CNPG_PASSWORD=<secret> CNPG_SSL_MODE=require.

## Context 48-hour follow-up after merging the DIRE proxmox-connector + identity-model fix series (merged 2026-06-11). Verify the merged fixes held under live traffic and apply the remaining (dry-run-validated) remediation. OpenSpec proposal: `openspec/changes/refactor-device-identity-reconciliation/` Memory: `project_dire_proxmox_identity`, `feedback_network_agnostic_identity`. ## Merged PRs (all in `staging`, code.carverauto.dev) - **#3573** — reconcile_duplicates atom-crash fix + `:maintenance` off web-ng - **#3574** — skip IP-less proxmox guests + honor rule `scope_value` in credential test - **#3575** — proxmox guest IP/MAC by configured NIC (not ClusterIP VIP) + **converge drifted plugin assignments** (connector revival) - **#3577** — agent↔device anchor on `agent_id` + chimera guard + behavioral IP-merge guard (distinct-MAC, network-agnostic) ## Verify at +48h (target 2026-06-13) 1. **Blob oscillation bounded**: `SELECT count(*) FROM platform.device_identifiers;` should stay ~150K (was 12.3M→149K), not climb. 2. **No new chimeras**: connected agents on a device with >1 distinct `armis_device_id` or >32 MACs should be ~0. 3. **Connector alive**: proxmox virtualization card refreshing; assignment reconcile not crashing. ## Remaining remediation to APPLY (validated dry-run only) Validated live via `.agents/skills/demo-cnpg-local-web-ng` (DB route 10.0.2.12:32040; creds `demo/serviceradar-db-credentials`). `agent-links` dry-run plans: - `k8s-agent`: **create** clean device (off the 60-MAC `FAKER-HP-WOR-25900` chimera `sr:65b08097`) - `agent-k8s-cp2-worker3`: **restore** off `sr:6a62a996` Apply with: `DireRemediation.run(mode: :execute, steps: ["agent-links"])` (writes a rollback NDJSON manifest). Re-run dry-run first to re-confirm. ## Open follow-ups (NOT yet fixed — new work) - **tonka01 bolt**: ~12 phantom "dusk" agents from agent-UID churn (`agent-dusk` → `agent-agent-dusk` [double-prefix bug] → `agent-dusk01` + transient `-NNNN` suffixes), each pinned to a stray device; `agent-dusk` bolts tonka01 (`sr:129bcb1f`). Needs a "exactly one agent record per agent — reap stale duplicates" remediation step (dry-run-first). Find the double-prefix source (the `inferred_agent_id` guards in falco/trivy are idempotent, so it's elsewhere). - **pve host duplicates**: each pve node has one device with `integration_id=proxmox:hypervisor:pveNN` but **no IP**, plus N bare-IP devices (no node id) from network discovery → disjoint namespaces, never merge. Fix: proxmox **node** enrichment must emit node IP/MAC (host-level version of #3575) so the node device merges with the IP-discovered records. ## How to validate `.agents/skills/demo-cnpg-local-web-ng/scripts/start-local-web-ng.sh`, or one-shot `mix run -e '<code>'` in `elixir/web-ng` with `CNPG_HOST=10.0.2.12 CNPG_PORT=32040 CNPG_DATABASE=serviceradar CNPG_USERNAME=serviceradar CNPG_PASSWORD=<secret> CNPG_SSL_MODE=require`.

~~mfreeman451 referenced this issue 2026-06-13 11:35:29 +00:00~~

fix: clear stale DIRE device agent links #3778

mfreeman451 commented

2026-06-13 11:35:45 +00:00

Author

Owner

Update: PR #3778 fixes the remaining code gap in the agent-links remediation.

What changed:

the remediation now also normalizes denormalized platform.ocsf_devices.agent_id ownership
each connected agent has one live target device carrying its agent_id
stale live devices that still carried that same agent_id are cleared to NULL
integration coverage now exercises stale device-agent links and verifies the second run is idempotent

Live demo remediation already ran with this branch:

initial patched dry-run found 4 stale live device links (1 for agent-k8s-cp3-worker3, 3 for k8s-agent, including the old faker/chimera row)
execute completed with errors: 0
rollback manifest: /tmp/dire_agent_links_device_agent_links_1781350430.ndjson (4 rows)
post-run duplicate live ocsf_devices.agent_id owners: 0
post-run connected-agent chimera links: 0
device_identifiers count unchanged at 149470
Proxmox connector evidence remains fresh: 12 Proxmox devices modified in the last 24h, latest 2026-06-13 11:30:57, 115 total Proxmox-sourced devices
final dry-run is idempotent: 14 keep, 0 create/adopt/restore, 0 identifier moves, 0 stale device-agent links, 0 alias work

Leaving this issue open until #3778 merges; the PR body references Fixes #3578 so it should close on merge.

Update: PR #3778 fixes the remaining code gap in the `agent-links` remediation. What changed: - the remediation now also normalizes denormalized `platform.ocsf_devices.agent_id` ownership - each connected agent has one live target device carrying its `agent_id` - stale live devices that still carried that same `agent_id` are cleared to `NULL` - integration coverage now exercises stale device-agent links and verifies the second run is idempotent Live demo remediation already ran with this branch: - initial patched dry-run found 4 stale live device links (1 for `agent-k8s-cp3-worker3`, 3 for `k8s-agent`, including the old faker/chimera row) - execute completed with `errors: 0` - rollback manifest: `/tmp/dire_agent_links_device_agent_links_1781350430.ndjson` (4 rows) - post-run duplicate live `ocsf_devices.agent_id` owners: 0 - post-run connected-agent chimera links: 0 - `device_identifiers` count unchanged at 149470 - Proxmox connector evidence remains fresh: 12 Proxmox devices modified in the last 24h, latest `2026-06-13 11:30:57`, 115 total Proxmox-sourced devices - final dry-run is idempotent: 14 keep, 0 create/adopt/restore, 0 identifier moves, 0 stale device-agent links, 0 alias work Leaving this issue open until #3778 merges; the PR body references `Fixes #3578` so it should close on merge.

mfreeman451 referenced this issue

2026-06-13 12:04:55 +00:00

fix: clear stale DIRE device agent links #3778

mfreeman451 commented

2026-06-13 12:05:43 +00:00

Author

Owner

Update: PR #3778 has been merged to staging for the validated agent-link cleanup. It added the DIRE remediation guard for one live ocsf_devices.agent_id owner per connected agent and cleared the stale live device agent_id links in demo.

Post-merge live checks still show #3578 should remain open: duplicate live agent_id owners are now 0, but the issue body's remaining follow-ups are still present in demo: stale dusk-* / agent-agent-dusk-* duplicate devices, and pve02/pve03/pve04 Proxmox hypervisor records still split from IP-discovered host records.

Update: PR #3778 has been merged to `staging` for the validated agent-link cleanup. It added the DIRE remediation guard for one live `ocsf_devices.agent_id` owner per connected agent and cleared the stale live device `agent_id` links in demo. Post-merge live checks still show #3578 should remain open: duplicate live `agent_id` owners are now 0, but the issue body's remaining follow-ups are still present in demo: stale `dusk-*` / `agent-agent-dusk-*` duplicate devices, and pve02/pve03/pve04 Proxmox hypervisor records still split from IP-discovered host records.

mfreeman451 referenced this issue

2026-06-13 12:17:28 +00:00

fix: retire stale DIRE agent-owned devices #3780

mfreeman451 commented

2026-06-13 12:17:38 +00:00

Author

Owner

Opened PR #3780 for the stale agent UID churn portion of this issue.

What it fixes:

adds a stale-agent-devices DIRE remediation step
targets unavailable historical agents matching the known stale dusk exact IDs/prefixes before a cutoff
removes stale agent_id identifiers and alias state residue
soft-deletes only devices directly owned by those stale agents when no active agent links to that device
hard-deletes the stale unavailable agent rows and writes rollback manifest records

Validation on the PR branch:

format check passed for touched files
scratch CNPG migrations completed
MIX_ENV=test mix test --include integration test/serviceradar/inventory/remediation/dire_remediation_test.exs passed: 6 tests, 0 failures
MIX_ENV=test mix compile --warnings-as-errors passed

Leaving #3578 open after #3780 because the remaining Proxmox node enrichment / bare-IP split is still not fixed by this PR.

Opened PR #3780 for the stale agent UID churn portion of this issue. What it fixes: - adds a `stale-agent-devices` DIRE remediation step - targets unavailable historical agents matching the known stale `dusk` exact IDs/prefixes before a cutoff - removes stale `agent_id` identifiers and alias state residue - soft-deletes only devices directly owned by those stale agents when no active agent links to that device - hard-deletes the stale unavailable agent rows and writes rollback manifest records Validation on the PR branch: - format check passed for touched files - scratch CNPG migrations completed - `MIX_ENV=test mix test --include integration test/serviceradar/inventory/remediation/dire_remediation_test.exs` passed: 6 tests, 0 failures - `MIX_ENV=test mix compile --warnings-as-errors` passed Leaving #3578 open after #3780 because the remaining Proxmox node enrichment / bare-IP split is still not fixed by this PR.

mfreeman451 referenced this issue

2026-06-13 12:26:21 +00:00

fix: preserve Proxmox host NIC identity #3781

mfreeman451 commented

2026-06-13 12:26:34 +00:00

Author

Owner

Follow-up PR #3781 is open for the Proxmox node identity part of this issue.

What changed:

The Proxmox plugin now preserves node-network hwaddr from /nodes/{node}/network.
The normalized Proxmox details JSON emits that host NIC identity as mac_address.
The Elixir Proxmox ingestor maps host NIC mac_address plus host interface IP evidence into provider-neutral virtualization network interface records.
This feeds the already-existing HypervisorEnrichmentIngestor host-MAC DIRE resolution path, so Proxmox hypervisor hosts can merge onto existing discovered devices instead of creating parallel placeholders.

Validation:

cd go/cmd/wasm-plugins/proxmox && go test ./...
cd elixir/serviceradar_core && mix format --check-formatted lib/serviceradar/inventory/proxmox_enrichment_ingestor.ex test/serviceradar/inventory/proxmox_enrichment_ingestor_test.exs
cd elixir/serviceradar_core && MIX_ENV=test mix compile --warnings-as-errors
srql-fixtures scratch CNPG migrated cleanly and passed MIX_ENV=test mix test --include integration test/serviceradar/inventory/proxmox_enrichment_ingestor_test.exs test/serviceradar/inventory/hypervisor_enrichment_ingestor_db_test.exs:170

Keeping this issue open until #3780 and #3781 are reviewed/merged and the remaining demo DIRE check confirms the duplicate Proxmox/bare-IP shape is gone.

Follow-up PR #3781 is open for the Proxmox node identity part of this issue. What changed: - The Proxmox plugin now preserves node-network `hwaddr` from `/nodes/{node}/network`. - The normalized Proxmox details JSON emits that host NIC identity as `mac_address`. - The Elixir Proxmox ingestor maps host NIC `mac_address` plus host interface IP evidence into provider-neutral virtualization network interface records. - This feeds the already-existing `HypervisorEnrichmentIngestor` host-MAC DIRE resolution path, so Proxmox hypervisor hosts can merge onto existing discovered devices instead of creating parallel placeholders. Validation: - `cd go/cmd/wasm-plugins/proxmox && go test ./...` - `cd elixir/serviceradar_core && mix format --check-formatted lib/serviceradar/inventory/proxmox_enrichment_ingestor.ex test/serviceradar/inventory/proxmox_enrichment_ingestor_test.exs` - `cd elixir/serviceradar_core && MIX_ENV=test mix compile --warnings-as-errors` - srql-fixtures scratch CNPG migrated cleanly and passed `MIX_ENV=test mix test --include integration test/serviceradar/inventory/proxmox_enrichment_ingestor_test.exs test/serviceradar/inventory/hypervisor_enrichment_ingestor_db_test.exs:170` Keeping this issue open until #3780 and #3781 are reviewed/merged and the remaining demo DIRE check confirms the duplicate Proxmox/bare-IP shape is gone.

mfreeman451 referenced this issue from a commit

2026-06-13 12:38:00 +00:00

fix: retire stale DIRE agent-owned devices

mfreeman451 referenced this issue

2026-06-13 12:38:36 +00:00

fix: preserve Proxmox host NIC identity #3783

mfreeman451 referenced this issue from a commit

2026-06-13 12:39:01 +00:00

fix: preserve Proxmox host NIC identity

mfreeman451 commented

2026-06-13 12:39:43 +00:00

Author

Owner

#3578 update: the two remaining DIRE cleanup code slices are now merged into staging.

Merged fixes:

PR #3780: adds the stale-agent-devices remediation step for unavailable historical agent UID churn rows, including stale agent_id identifier cleanup, alias cleanup, guarded device soft-delete, and rollback manifest records.
PR #3783: preserves Proxmox host NIC identity by carrying node-network hwaddr through the Go plugin/normalized JSON and mapping host NIC MAC/IP evidence into the provider-neutral hypervisor enrichment path.

Validation already documented in the PRs:

#3780: format check, scratch CNPG migrations, DIRE remediation integration test file (6 tests, 0 failures), compile with warnings-as-errors.
#3783: Proxmox Go tests, Elixir format/compile, srql-fixtures scratch CNPG migrations plus Proxmox/hypervisor ingestor tests.

Keeping #3578 open until demo is rolled to staging containing #3780/#3783 and we verify the live checks: device_identifiers count remains bounded, no new chimeras, Proxmox virtualization/node cards refresh, stale dusk agent rows are gone, and Proxmox host bare-IP duplicates converge.

#3578 update: the two remaining DIRE cleanup code slices are now merged into staging. Merged fixes: - PR #3780: adds the stale-agent-devices remediation step for unavailable historical agent UID churn rows, including stale agent_id identifier cleanup, alias cleanup, guarded device soft-delete, and rollback manifest records. - PR #3783: preserves Proxmox host NIC identity by carrying node-network hwaddr through the Go plugin/normalized JSON and mapping host NIC MAC/IP evidence into the provider-neutral hypervisor enrichment path. Validation already documented in the PRs: - #3780: format check, scratch CNPG migrations, DIRE remediation integration test file (6 tests, 0 failures), compile with warnings-as-errors. - #3783: Proxmox Go tests, Elixir format/compile, srql-fixtures scratch CNPG migrations plus Proxmox/hypervisor ingestor tests. Keeping #3578 open until demo is rolled to staging containing #3780/#3783 and we verify the live checks: device_identifiers count remains bounded, no new chimeras, Proxmox virtualization/node cards refresh, stale dusk agent rows are gone, and Proxmox host bare-IP duplicates converge.

mfreeman451 referenced this issue

2026-06-13 14:45:46 +00:00

Handoff: resume demo rollout and E2E from staging 98b2f55 #3786