Case study
AI-Powered Security Review Pipeline
A multi-model security review pipeline that reads 1Password pull request diffs, grounded in five years of the team’s own review history and a compressed map of the codebase it’s reviewing.
The problem
Product Security reviewed pull requests by hand, on recurring calls. The reviews were thorough and they caught real issues, but PR volume kept climbing as the company grew and engineers adopted AI coding assistants. Those reviews cost the team countless hours every month, much of it spent on vulnerability patterns they had already seen many times. Manual review couldn’t keep pace with that kind of growth.
What I built
SAGE (Security Analysis Guidance Engine), an automated review pipeline that encodes the team’s institutional knowledge and applies it to PR diffs:
- 01 Ruleset: Nearly 9,000 PRs across five years and five languages (Rust, Go, Kotlin, TypeScript, Swift), deduplicated with LLMs into 8,343 clean reviews and distilled into a ruleset spanning 16 vulnerability categories. It grew from 171 rules at v0 to 343 at v1.
- 02 Three-stage adversarial pipeline: A fast, cost-efficient Finder optimized for recall; a Critic running a frontier reasoning model from a different vendor to attack each finding; and a Judge that renders the final verdict: confirmed, false positive, or needs review. Splitting providers keeps one model’s blind spots from becoming the pipeline’s.
- 03 Vendor-agnostic harness: A Go
llm.Clientinterface with per-stage model profiles, structured JSON output, and prompt injection detection, so models can be swapped as they improve without touching the pipeline. - 04 Delivery: A GitHub Action that posts line-specific comments directly on the PR, backed by an activity log that tracks scan history and false positives over time.
- 05 SuperSAGE context layer: 1Password monorepos are far too large for any context window, so a
map-reduce pass summarizes each directory into a
SCAFFOLDING.mdfile containing sensitivity ratings, attack surfaces, trust boundaries, and file summaries. Those summaries fold upward, child into parent, to the root. The result is roughly 30× smaller than the source it describes. - 06 Deterministic change detection: LLMs rarely produce the same text twice, and reworded summaries would cascade phantom diffs up the whole tree. So the Go harness hashes a small, fixed set of structural facts with SHA-256 (the files present, their sensitivity ratings, and trust-boundary designations) and compares only those. Prose changes are ignored; real structural changes aren’t. Only directories with genuine code changes are regenerated.
The result
On a cryptographic PR used as an early benchmark, SAGE surfaced 6/6 true positives, including issues human reviewers initially missed, at an average cost of $0.47 USD per scan. The context map refreshes every night for a few dollars, so reviews reason about a diff in light of the code around it rather than in isolation. It saves the Product Security team hours of review time every week, and the knowledge it applies no longer lives only in the heads of the people who happened to attend the call.