Skip to content

feat: drop-in observability kit with audit comparison and behavioral signals#22711

Merged
mnkiefer merged 19 commits intomainfrom
obs-tools
Mar 25, 2026
Merged

feat: drop-in observability kit with audit comparison and behavioral signals#22711
mnkiefer merged 19 commits intomainfrom
obs-tools

Conversation

@mnkiefer
Copy link
Contributor

@mnkiefer mnkiefer commented Mar 24, 2026

  • Adds the foundation of a drop-in observability kit that consumers can adopt via gh aw add or gh aw add-wizard to get recurring repo-wide reports on workflow health, drift, and risk.
  • This includes:
    • Audit comparison: finds a successful baseline run, computes deltas (turns, posture, blocked requests, MCP failures), and classifies changes as stable, changed, or risky.
    • Task domain and behavior fingerprinting: infers run type and builds a compact execution profile for cross-run comparison.
    • Agentic assessments: flags resource-heavy, weakly-controlled, or unnecessarily agentic runs.
    • Observability insights: per-run and fleet-level signals surfaced in audit and logs JSON output.
    • Deterministic execution episodes and lineage edges: gh aw logs now emits structured episodes[] and edges[] data so orchestrated runs can be grouped and rolled up consistently.
  • The bundled workflow now follows the enterprise-style reporting model of one full discussion report per run, plus at most one escalation issue when the findings warrant action.
  • Opt-in job summary remains available via frontmatter for in-run visibility:
observability:
  job-summary: on

Copilot AI review requested due to automatic review settings March 24, 2026 14:51

This comment was marked as outdated.

@mnkiefer mnkiefer self-assigned this Mar 24, 2026
@pelikhan
Copy link
Contributor

@copilot review comments, lint go

Copy link
Contributor

Copilot AI commented Mar 24, 2026

@copilot review comments, lint go

Addressed all review comments in commit 5abdb23:

  1. countBlockedRequests() — now sums across all gatewayEventPaths files instead of returning after the first match.
  2. Audit comparison classification — added reason codes for beneficial changes (turns_decrease, blocked_requests_decrease, mcp_failures_resolved) and explicit switch cases so runs that differ from baseline are no longer classified as stable.
  3. renderObservabilityInsights — added critical and low severity icons.
  4. min_blocked_requests evidence — now includes both threshold and actual count, e.g. blocked_requests_gte=5 actual=7.
  5. warn action — added explicit case "warn" using FormatWarningMessage so it renders distinctly from info messages.

Copilot AI requested a review from pelikhan March 24, 2026 18:06
@mnkiefer mnkiefer changed the title feat: improve agentic audit baselines and execution observability feat: strengthen agentic audit baselines and execution observability Mar 24, 2026
@pelikhan pelikhan marked this pull request as ready for review March 24, 2026 20:00
@github-actions github-actions bot added the lgtm label Mar 24, 2026
@github-actions

This comment has been minimized.

@github-actions
Copy link
Contributor

Hey @mnkiefer 👋 — great work on this observability overhaul! Strengthening gh aw audit with baseline comparisons, behavioral fingerprinting, and agentic assessments, while also levelling up gh aw logs with trend-aware reporting, is a really cohesive and valuable addition to the toolkit. The drop-in observability workflow kit is a particularly nice touch for teams who want to operationalize monitoring across repos without starting from scratch.

The PR is well-structured, thoroughly described, and backed by solid test coverage across all the new modules. It looks ready for maintainer review. 🟢

Generated by Contribution Check ·

@mnkiefer mnkiefer changed the title feat: strengthen agentic audit baselines and execution observability feat: drop-in observability kit with audit comparison and behavioral signals Mar 25, 2026
@mnkiefer mnkiefer merged commit eb35bd9 into main Mar 25, 2026
109 checks passed
@mnkiefer mnkiefer deleted the obs-tools branch March 25, 2026 08:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants