AI agent hooks for secrets, scoring, and developer experience

Internal system names, providers, and exact numbers have been abstracted or generalized for confidentiality — the architecture patterns and trade-offs described are accurate.

Context

The team uses an LLM coding agent extensively — for code review, pair-programming, ops tasks. Once the agent stops being a toy and becomes part of daily work, two problems show up:

1. Sensitive data leaks into context. Tool output (database queries, environment dumps, log slices) routinely contains tokens, keys, and customer-identifying fields. A naive setup pipes that straight into the model and into chat history.
2. Repeated workflows stay inconsistent. Imports from a prod DB dump, dropping into an ECS shell, running a parameterized query — each engineer reinvents the recipe slightly differently.

The agent platform exposes hooks at well-defined points (user prompt submission, tool invocation, tool output) and supports custom slash commands and skills. The work was to use those hooks to harden the agent for production use and standardize ops workflows behind one-line commands.

Architecture

User prompt ──► UserPromptSubmit hook ──► (redact secrets in prose)
                                            │
                                            ▼
Tool call (Bash / API / file) ──► tool runs
                                       │
                                       ▼
                          PostToolUse:Bash hook ──► (redact tool output)
                                       │
                                       ▼
                              Model receives clean context

Slash commands ──► /db-import, /sql-prod, /ecs-shell, /act-on-pr, /create-pr
Skills          ──► reusable agent capabilities (act-on-pr, create-pr, ...)
PR Size hook    ──► weighted scoring + dev-experience signal label

Three-layer secret redaction

A single regex pass is not enough; secrets come into context in three different shapes:

• User-typed prose. Engineers paste tokens into the prompt accidentally — "use this token: eyJ…". Caught at the UserPromptSubmit hook, before the model sees it.
• Tool stdout. A psql query or an aws call dumps data that may contain keys, OAuth tokens, customer documents. Caught at PostToolUse:Bash — the agent sees a redacted version, never the raw output.
• Codebase rules. The agent itself has a guideline never to echo secrets back in narrative responses, even if it sees them. This is a behavior rule, not a hook, but it sits in the same redaction strategy and catches cases the hooks miss.

The hook started in dry-run mode (logging what would be redacted) and then pivoted to block mode once the false-positive rate was acceptable. That two-step rollout meant nobody got blocked by a buggy regex on day one.

Weighted PR-size scoring

PR size labels (S/M/L/XL) were originally line-count based, which mis-classifies migrations and generated files as huge changes when they're trivial to review. The scoring was reworked to weight changes by file type and to surface a separate dev-experience signal label when the diff touches tooling/CI rather than product code. Adapted from a sibling backend's implementation.

Slash commands as ops surface

Operational workflows that engineers ran ad-hoc became one-line commands:

• /db-import — robust DB import that survives the mysql:8 init race and bumps packet/row limits automatically.
• /sql-prod — parameterized read-only query against prod with audit logging.
• /ecs-shell — open a shell on an ECS task without remembering the cluster/service/task triplet.
• /act-on-pr, /create-pr — wrap the PR workflows the team converged on.

Same surface for everyone — the recipe ships in the repo, not in someone's head.

Trade-offs

Block mode for secret redaction, not log-and-warn. Block mode means a buggy regex can interrupt legitimate output. The benefit is that the day one secret slip never happens — a leaked credential in chat history is a credential rotation, not a "we'll fix the warning later." The mitigation was the dry-run period before flipping the switch.

Three layers, not one. More moving parts, more places to keep aligned. The benefit is defense in depth: a secret that slips past one layer gets caught by the next. Each layer is also independent — disabling the tool-output redactor doesn't disable the input one.

Slash commands as repo artifacts, not personal aliases. Engineers used to have shell aliases; everyone slightly different. Codifying them in the repo means changes have to land via PR. The benefit is that onboarding stops being "ask Alice for her dotfiles" — the ops surface is shared and reviewable.

Weighted PR-size scoring, not line count. Maintenance cost: the weight table needs to evolve with the codebase. The benefit is that the team stopped getting "XL" labels on PRs that move a migration up or down by one digit, and started getting them on actual large changes that deserve more reviewers.

Outcome

• Secret hygiene. Three independent layers of redaction; the chat history and tool outputs the model sees are stripped of credentials before they land.
• Operational consistency. Recipes that lived in tribal knowledge — DB imports, prod read queries, ECS shells — became one-line commands available to everyone.
• PR scoring people trust. Labels actually reflect review effort, separating product-code volume from tooling/CI changes.
• Repeatable hardening pattern. The combination of dry-run → block → measure became the team's default approach for any new agent guardrail.