carverauto/serviceradar

Fork 0

feat(flow-templates): inspect NetFlow/IPFIX templates in web UI #3243

Open

mikemiles-dev wants to merge 5 commits from mikemiles-dev/flow-templates-ui into staging AGit

mikemiles-dev commented

2026-05-08 04:09:55 +00:00

Collaborator

Why: confirms multi-replica template propagation through the KV bucket
is working in deployed clusters and gives ops a way to see what each
router is sending without shelling into a collector pod.

Surfaces:

Standalone page: /observability/flow-templates
Inline tab card: /observability?tab=netflows&view=templates (5th
card alongside Traffic Analysis, Talkers, Topology, Flow Explorer)
JSON API: GET /api/flow-templates and /api/flow-templates/:scope

Plumbing:

proto/kv.proto: optional bucket field on Get/BatchGet/ListKeys/Info
so datasvc clients can read non-default KV buckets.
go/pkg/datasvc: new GetEntryFromBucket / ListKeysInBucket interface
methods, read-only and require the bucket to already exist (datasvc
does not auto-create non-default buckets). nats_security mode "none"
no longer requires a creds file (was rejecting plain unauth NATS dev
setups).
elixir/datasvc: bucket: opt on KV.list_keys/2, get/2, info/1.
elixir/web-ng: pure-Elixir wire-format decoder for the netflow_parser
template payloads (V9 / V9 options / IPFIX / IPFIX options + V9-style
IPFIX), curated v9 + IPFIX field-name registry copied from the rust
netflow_parser IANA enums (~158 v9 + ~500 IPFIX entries with
field_NNN / enterprise_EEE_field_NNN fallbacks).
rust/kvutil: GetRequest literal updated for the new bucket field.
rust/flow-collector: dev config flips on metrics_addr + template_store
by default; Dockerfile pinned to rust:1.88-bookworm so produced
binaries match the bookworm-slim runtime GLIBC.
docker/compose: simple-compose dev stack lands as tracked files
(datasvc.simple.json + the relaxed validation), and the web-ng
Dockerfile copies elixir_uuid which was a previously-missing path
dep.

Tests: parser, decoder (round-trips rust encoder layouts incl
enterprise + scope fields), API controller error paths.

Adds an operator-facing view of NetFlow v9 / IPFIX templates the flow-collector replicas have learned, decoded from the shared flow_templates NATS KV bucket and grouped per exporter (router) with per-template field lists. Why: confirms multi-replica template propagation through the KV bucket is working in deployed clusters and gives ops a way to see what each router is sending without shelling into a collector pod. Surfaces: - Standalone page: /observability/flow-templates - Inline tab card: /observability?tab=netflows&view=templates (5th card alongside Traffic Analysis, Talkers, Topology, Flow Explorer) - JSON API: GET /api/flow-templates and /api/flow-templates/:scope Plumbing: - proto/kv.proto: optional bucket field on Get/BatchGet/ListKeys/Info so datasvc clients can read non-default KV buckets. - go/pkg/datasvc: new GetEntryFromBucket / ListKeysInBucket interface methods, read-only and require the bucket to already exist (datasvc does not auto-create non-default buckets). nats_security mode "none" no longer requires a creds file (was rejecting plain unauth NATS dev setups). - elixir/datasvc: bucket: opt on KV.list_keys/2, get/2, info/1. - elixir/web-ng: pure-Elixir wire-format decoder for the netflow_parser template payloads (V9 / V9 options / IPFIX / IPFIX options + V9-style IPFIX), curated v9 + IPFIX field-name registry copied from the rust netflow_parser IANA enums (~158 v9 + ~500 IPFIX entries with field_NNN / enterprise_EEE_field_NNN fallbacks). - rust/kvutil: GetRequest literal updated for the new bucket field. - rust/flow-collector: dev config flips on metrics_addr + template_store by default; Dockerfile pinned to rust:1.88-bookworm so produced binaries match the bookworm-slim runtime GLIBC. - docker/compose: simple-compose dev stack lands as tracked files (datasvc.simple.json + the relaxed validation), and the web-ng Dockerfile copies elixir_uuid which was a previously-missing path dep. Tests: parser, decoder (round-trips rust encoder layouts incl enterprise + scope fields), API controller error paths.

mikemiles-dev requested review from mfreeman451

2026-05-08 04:09:55 +00:00

mikemiles-dev force-pushed mikemiles-dev/flow-templates-ui from 7089194ad1

lint / lint (pull_request) Blocked by required conditions

Details

CI / build (pull_request) Blocked by required conditions

Details

Secret Scan / gitleaks (pull_request) Blocked by required conditions

Details

to 1c47a0eaff

Secret Scan / gitleaks (pull_request) Successful in 21s

Details

lint / lint (pull_request) Failing after 1m26s

Details

CI / build (pull_request) Has been cancelled

Details

2026-05-08 04:15:33 +00:00

Compare

mikemiles-dev force-pushed mikemiles-dev/flow-templates-ui from 3cbbdc9f5a

Secret Scan / gitleaks (pull_request) Successful in 21s

Details

lint / lint (pull_request) Failing after 1m25s

Details

CI / build (pull_request) Failing after 5m50s

Details

to d8ebafe923

Secret Scan / gitleaks (pull_request) Successful in 23s

Details

lint / lint (pull_request) Successful in 1m9s

Details

CI / build (pull_request) Failing after 5m6s

Details

2026-05-09 15:11:59 +00:00

Compare

mfreeman451 reviewed

2026-05-11 02:53:35 +00:00

mfreeman451 left a comment

lgtm

Secret Scan / gitleaks (pull_request) Successful in 23s

Details

lint / lint (pull_request) Successful in 1m9s

Details

CI / build (pull_request) Failing after 5m6s

Details

This pull request has changes conflicting with the target branch.

Cargo.lock
elixir/web-ng/lib/serviceradar_web_ng_web/router.ex
k8s/demo/base/serviceradar-flow-collector.yaml
proto/kv.pb.go
proto/kv_grpc.pb.go
rust/flow-collector/Cargo.toml
rust/flow-collector/src/config.rs
rust/flow-collector/src/main.rs
rust/flow-collector/src/metrics.rs
rust/flow-collector/src/netflow/mod.rs
rust/flow-collector/src/publisher.rs

View command line instructions

Manual merge helper

Use this merge commit message when completing the merge manually.

Merge commit title

Merge pull request 'feat(flow-templates): inspect NetFlow/IPFIX templates in web UI' (#3243) from mikemiles-dev/flow-templates-ui into staging

Merge commit body

Reviewed-on: https://code.carverauto.dev/carverauto/serviceradar/pulls/3243

Merge pull request 'feat(flow-templates): inspect NetFlow/IPFIX templates in web UI' (#3243) from mikemiles-dev/flow-templates-ui into staging

Reviewed-on: https://code.carverauto.dev/carverauto/serviceradar/pulls/3243

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin +refs/pull/3243/head:mikemiles-dev/flow-templates-ui

git switch mikemiles-dev/flow-templates-ui

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch staging

git merge --no-ff mikemiles-dev/flow-templates-ui

git switch mikemiles-dev/flow-templates-ui

git rebase staging

git switch staging

git merge --ff-only mikemiles-dev/flow-templates-ui

git switch mikemiles-dev/flow-templates-ui

git rebase staging

git switch staging

git merge --no-ff mikemiles-dev/flow-templates-ui

git switch staging

git merge --squash mikemiles-dev/flow-templates-ui