How do I log and audit what an AI tool does in my repos?

How do I log and audit what an AI tool does in my repos?

Core audit features (non-negotiables)

• PR-embedded comments with context. Each suggestion/flag/fix names the file & line, the finding, and the reason (rule or logic), stamped with time and actor (e.g., @optibot).
• Audit trail lives in-repo. Evidence sits in the PR thread and/or GitHub Checks—versioned, searchable, and available long after a vendor UI session expires.
• Version-controlled summary (recommended). Emit a concise /audit/*.md file per PR or release (rules run, outcomes, links to comments). Auditors and leads can skim one canonical record.

Minimal fields your trail must capture

Where each action should be logged

“Reasoning drawer” (evidence, not vibes)

Every artifact should expose why the AI acted—briefly and repeatably:

• Finding: Hard-coded token
• File: config/__init__.py:87
• Reason: Secret pattern match (AWS_KEY v3), not present in allow-list
• Evidence: AKIA****************
• Action: Suggested removal + env var replacement
• Actor: @optibot • 2025-08-16T14:22:11Z

Prefer deterministic rule IDs (e.g., SEC-SECRET-003, PII-EMAIL-001) over fuzzy prose. Link to the specific diff hunk—one click from claim to code.

Optional: per-PR summary file pattern

Create a lightweight, machine-and-human-readable summary the bot updates once per run:

/audit/optibot-checks.md

# PR Audit Summary

**PR:** #4821 • **Commit:** 9e1b4c2  
**Run:** 2025-08-16T14:22Z • **Actor:** @optibot

## Checks
- SEC-SECRET-003 (secrets): 1 found → fix proposed
- PII-EMAIL-001 (PII): 0
- CI-BUILD-101 (workflow): pass

## Findings
- [config/__init__.py:87] Hard-coded token → see comment
 - Disposition: Fixed via PR #4822 (awaiting review)

## Notes
- Branch protections enforced; no direct writes to `main`.

Benefits: one file to skim, diffable across reruns, and audit-friendly for SOC2/ISO evidence.

Safe rollout (noise-free)

• Start with Checks, not comments. One GitHub Check that links to detailed evidence; promote to PR comments only for must-fix items.
• Gate “write” features. Keep read-only by default. Enable “open fix PR” in config for specific repos/branches (e.g., bot/fixes/*) with required human approval.
• Config, not trust. A repo config (e.g., .optibot.yml) should toggle: comment vs check, must-fix categories, and allowed branches. If the feature is off, the permission disappears.

Mini-case: “Ship day, not post-mortem”

A frontend PR introduces a token in config/__init__.py.
The reviewer bot posts a must-fix comment with rule ID + evidence, fails the Checks status, and opens a scoped fix PR that swaps the token for an env var.
Release notes and /audit/optibot-checks.md capture the full trail.
Outcome: secret never hits main, audit passes with receipts.

Try it safely

Run the reviewer read-only first: Checks-only, must-fix comments, summary file.
If you want fixes, allow bot branches and require human approval.
You keep control; the agent earns privileges—with receipts.