CI checks

npx skills add tylerdavis/graph

jobs:
  docs-drift:
    runs-on: ubuntu-24.04
    container:
      image: ghcr.io/tylerdavis/graph:v0.11.0      # graph + git/jq/gh baked in
      credentials:
        username: ${{ github.actor }}
        password: ${{ secrets.GITHUB_TOKEN }}
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }                   # plans diff refs locally
      - run: graph plan run docs_drift --input base="$BASE" --input head="$HEAD"

​

The pieces

​

docs_drift — an inferred gate as a merge check

steps:
  - id: E0
    tool_name: builtin__git_changed_files
    input: { base: "{{input.base}}", head: "{{input.head}}", prefix: "crates/" }
  - id: E1
    tool_name: exit
    input:
      when: { value: "{{E0.count}}", op: eq, to: 0 }
      status: success
      message: no crate code changed — docs gate not applicable
      output: { docs_required: false, reason: no_crate_changes }
  - id: E2
    tool_name: builtin__git_changed_files
    input: { base: "{{input.base}}", head: "{{input.head}}", prefix: "docs/" }
  - id: E3
    tool_name: exit
    input:
      when: { value: "{{E2.count}}", op: gt, to: 0 }
      status: success
      message: docs updated alongside crate changes
      output: { docs_required: false, reason: docs_touched }
  - id: E4
    tool_name: builtin__git_diff
    input: { base: "{{input.base}}", head: "{{input.head}}",
             paths: "crates/", exclude: "", max_bytes: 150000 }
  - id: E5
    tool_name: exit
    input:
      infer: |
        Does this diff change user-visible behavior of the graph CLI …
        Pure refactors, comment-only changes, test-only changes, and
        internal renames do not count. Answer yes only if the behavior a
        user or plan author observes would change.

        {{E4.text}}
      status: error
      message: >-
        behavioral change without a docs/ update — CLAUDE.md requires docs
        land in the same PR as the behavior change
output:
  docs_required: false
  reason: "{{E5.reason}}"

• Short-circuit before you infer. E1 and E3 are when gates over github pack tool output — a docs-only PR exits success in milliseconds with zero LLM calls. The infer gate is the last resort, and it sees only the truncated crates/ diff. Worst case per PR: one judge call on the cheap judge role.
• The gate is the CI contract. status: error maps to process exit code 4 — distinct from exit 1 (the plan itself broke). The job needs no branching: exit 4 fails the step, and under GRAPH_EVENTS=github graph prints the gate’s message as a ::error:: annotation itself.
• Every exit path emits the same shape. Both success gates and the fall-through output produce {docs_required, reason} — callers parse one contract. On a non-fired infer gate, {{E5.reason}} carries the judge’s reasoning into the output for free (the judge explains why no docs are needed).
• Ask the judge a falsifiable question. The prompt defines what does not count (refactors, test-only changes). The judge role answers “yes only when the data clearly supports it”, so the framing sets the false-positive rate. Live calibration: it flagged a commit adding an env var and a JSON field (“observable by users and plan authors”), and passed a dead-code removal with its reasoning in reason.

​

graph_review_9000 — a composed reviewer

• graph_review_core — the review engine, purely local (git only). Fetches the changed files, diff, and full file contents for a change set; clusters the diff into functional groups; scouts each scoutworthy group’s blast radius with bounded agent steps; runs the eight-pass rubric; returns structured JSON (output: finish — framing, a severity-ordered pass table, findings). It takes previously known findings as a suppression list, so callers with history get only what’s new.
• graph_review_thread_audit — the thread lifecycle, GitHub-coupled. Given the prior findings read back from PR review threads, it verifies fixes against the file contents at head and applies the transitions (fixed → resolved with a reply, dismissed by a human reply → declined; insufficient evidence keeps a finding open).
• graph_review_9000 — the wrapper CI runs. PR metadata, thread readback, a decide gate that calls the audit only when prior findings exist, the core review, and the posting — inline threads for new findings, a marker-keyed summary comment that refreshes in place. Open threads and human replies are never deleted.

steps:
  - id: E0
    tool_name: builtin__gh_pr_meta
    input: { pr: "{{input.pr}}" }
  - id: E2
    tool_name: builtin__git_diff             # gate-only: is there anything to review?
    input: { base: "{{E0.base_sha}}", head: "{{E0.head_sha}}",
             paths: ".", exclude: "Cargo.lock CHANGELOG.md", max_bytes: 1024 }
  - id: E4b
    tool_name: builtin__gh_pr_review_threads # prior findings + thread states
    input: { pr: "{{input.pr}}", marker: "graph_review_9000:finding" }
  - id: G1
    tool_name: decide                        # audit only when history exists
    input:
      if: { value: "{{E4b.has_findings}}", op: eq, to: true }
      then:
        tool_name: plan__graph_review_thread_audit
        input: { pr: "{{input.pr}}", base: "{{E0.base_sha}}", head: "{{E0.head_sha}}",
                 title: "{{E0.title}}", findings: "{{E4b.findings}}" }
      else:
        tool_name: builtin__reshape          # the audit's output contract, empty
        input:
          shape: { findings: [], threads: [], still_open_count: 0,
                   resolved_now_count: 0, declined_now_count: 0,
                   transitions_count: 0, has_transitions: false }
  - id: E3
    tool_name: exit                          # after thread hygiene, so reverts settle
    input:
      when: { value: "{{E2.text}}", op: empty }
      status: success
      message: nothing to review
  - id: C1
    tool_name: plan__graph_review_core       # the engine; returns structured JSON
    input: { base: "{{E0.base_sha}}", head: "{{E0.head_sha}}",
             title: "{{E0.title}}", body: "{{E0.body}}",
             known_findings: "{{G1.result.findings}}" }
  - id: E6
    tool_name: map                           # findings -> comment bodies + payloads
    input:
      over: "{{C1.findings}}"
      concurrency: 8
      do:
        tool_name: builtin__reshape
        input:
          shape:
            path: "{{item.file}}"
            line: "{{item.line}}"
            payload: { label: "{{item.label}}", severity: "{{item.severity}}", ... }
            body: |
              > [!{{item.alert_type}}]
              > **{{item.severity_label}}: {{item.finding}}**
              > {{item.explanation}}
  - id: E7
    tool_name: builtin__gh_pr_thread_sync    # creates only; the audit did transitions
    input:
      pr: "{{input.pr}}"
      head_sha: "{{E0.head_sha}}"
      marker: "graph_review_9000:finding"
      creates: "{{E6.results}}"
      threads: []
  - id: E8
    tool_name: builtin__gh_pr_comment
    input:
      pr: "{{input.pr}}"
      marker: "graph_review_9000:summary"
      body: |
        > [!{{C1.overall_alert_type}}]
        > **graph_review_9000** · {{C1.overall_framing}}

        | Pass | Area | Status |
        |---|---|---|
        {{#C1.passes}}| {{number}} | {{area}} | {{status_emoji}} {{status_label}} |
        {{/C1.passes}}

        🧵 {{E7.created_count}} new · {{G1.result.still_open_count}} open ·
        {{G1.result.resolved_now_count}} resolved · {{G1.result.declined_now_count}} declined

​

The same engine, locally

# review the working tree against main (stage new files first: git add -N .)
graph plan run graph_review_core --input base=main

# review a committed range
graph plan run graph_review_core --input base=origin/main --input head=HEAD

• The rubric is the product. The infer instruction enumerates this repo’s invariants pass by pass and instructs “omit findings that found nothing to flag”. A focused reviewer beats a generic one precisely because it refuses to comment on everything else. Its predecessor caught real bugs in this repo’s own PRs — including one in the PR that changed the reviewer itself.
• The rubric can live in the plan. builtin__infer accepts both the instruction and a caller-supplied output_schema, so the whole rubric is one step inside graph_review_core with no prompt tool to keep in sync. (The retired pr_review split them into a ./.graph/tools/ prompt tool — still the right shape when several plans share one rubric.)
• Sub-plans return data; branches must agree on a contract. A called plan with an output: finish hands its rendered JSON straight to the caller’s step — {{C1.findings}} is the core’s findings array, no unwrapping (composition). The decide’s two branches exploit the flip side: everything downstream reads {{G1.result.*}}, so the else-branch reshape emits the audit plan’s exact output keys as empty literals — the rejoined plan can’t tell which branch ran, which is the whole point of branching.
• Gate expensive history work on whether history exists. A first review has no threads to audit, but the audit’s prompt carries the full diff and file contents — review-sized money for a guaranteed-empty answer. The decide on {{E4b.has_findings}} makes the skip structural: a first-run PR pays for the review and nothing else.
• Presentation is a template, not code. builtin__reshape projects each finding into its comment body and the summary renders with sections — iteration builds the pass table, inverted sections handle the clean case. Logic stays in steps; the markdown is pure presentation.
• Ordering is asked for, not computed. The template dialect can’t sort, so the schema asks the model for blocker_ordinal and a severity-ordered passes array, and the templates iterate in the order they arrive. Logic belongs in the steps or the schema — never the markdown.
• Declare the output schema — it’s enforced. The schema pins passes to exactly eight (minItems/maxItems) and marks every finding field required. Models occasionally omit required fields even under provider-forced structured output; graph validates the result against the declared schema and runs one repair-role fix-up, so {{E5.overall_framing}} can never be a missing key at render time.
• Review the whole diff. The diff is scoped to nothing (paths: ".") minus generated files — an early version reviewed only crates/ and docs/, which blinded it to the highest-leverage changes in an infra PR: workflows, plans, and the exec tools they run.
• Ground absence claims in the complete file list, never the diff. The diff is byte-capped and git orders hunks by path, so on a big PR the docs/ hunks are exactly what falls off the end — an early version reported “docs not updated” false positives on a PR whose docs edits sat past the truncation point. The fix is layered, inside the core: repo_changed_files (E1) feeds the reviewer a complete, never-truncated file list, the filter (E3b) drops deleted entries whose contents can’t exist in the reviewed state — so the map over repo_file (E4) reads whole files rather than hunks without dying on a deletion, and its results stay index-aligned with {{E3b.items}} by construction — repo_diff appends a visible [diff truncated: …] marker when it cuts, and the rubric judges docs-parity and test-parity only from the complete list. Per-file truncation is the same trap one level down: the first run of the split reviewer asserted two output keys were missing from a 42KB plan file fetched under a 30KB budget — the keys sat past the cut, behind a visible [file truncated: …] marker. Hence a corollary in the finding bar (truncated evidence can never support an absence claim — ‘unverified’ at most, and the audit may not resolve or hold a finding on it either) and a per-file budget sized so the repo’s largest reviewed files arrive whole.
• Ask the model to refute itself. The instruction closes with a gate: try to disprove each blocker against the diff and demote what doesn’t survive, and mark a pass unverified rather than pass for anything you couldn’t check. A false clear costs as much as a false blocker.
• Findings flow through the pipeline, enriched at each step. reshape (E6) adds path/body/payload to each finding without dropping the original fields; gh_pr_thread_sync (E7) passes those extras through and adds url where a comment posted. So the blockers list (E8) renders from one array — severity and label from the review, the link from the post — with no join logic, which the template dialect couldn’t express anyway. Findings the model couldn’t pin to a line (or whose line the API rejects) simply come back without url, and the inverted section renders them as plain text.
• Findings are threads with a lifecycle, not comments to repost. Each finding’s comment embeds a hidden JSON payload (its label, severity, file, line, headline); gh_pr_review_threads (E4b) reads them back next run with thread states — open, resolved, declined (a human closed or dismissed it). Open threads and human replies are never deleted, so a conversation on a finding survives every push.
• Resolution is judged against the code, not against re-detection. The audit plan transitions an open finding only when the code at head shows the fix, and treats insufficient evidence as “stays open” — if it instead resolved whatever the reviewer failed to re-find, one flaky review pass would silently close real findings. Declines come only from explicit human replies, never from silence. The core’s rubric then sees the audited findings purely as a suppression list (known_findings): report only what’s new.
• The file contents at head are the resolution evidence, not the diff. A fix that lands in a later commit of the same PR leaves no hunk in the cumulative base-to-head diff — the flagged region simply stops appearing — so the audit is told to judge from the finding’s location in the full file contents. Live calibration: the first mixed-state run kept a fixed finding open precisely because the instruction let the model wait for a diff hunk that could never exist.
• Blast radius is scouted, not guessed — one scout per functional change. The diff and changed-file contents can’t show who calls what changed. Inside the core, an inference (E4e) clusters the diff into functional groups (newline-joined file_list/symbol_list strings keep the per-group templates logic-less), then a map (E4f) runs an agent step per group — a bounded loop over read-only repo_grep/repo_file chasing that group’s callers, implementations, and tests in the reviewed state — up to six in parallel, each returning a dossier that echoes its group name. The chain sits after the empty-diff gate (it only earns its cost when there’s something to review), and E5 renders each dossier behind a {{#final}} guard, so a scout that exhausts its budget costs its group’s evidence rather than the run. Focused scouts dig deeper than one generalist — deep enough that the round budget, not curiosity, has to be the limit.
• Scouts are dispatched only where code changed. A prose-only group — prompts, docs, configuration — has nothing a repository search can find, and a scout pointed at one will grind its whole round budget grepping fragments of English (live calibration: a prompt-only PR burned 60 rounds finding nothing before this guard existed). So the grouping call marks each group scoutworthy, a filter (E4e2) drops the rest before the map spends a round, and E5 sees which groups went deliberately unscouted via the filter’s dropped half — selection that narrows what runs next without losing what is known. The scout prompt keeps a bail-immediately instruction as the backstop for grouping misjudgments.
• Thread hygiene runs before the nothing-to-review gate. The readback (E4b) and the decide-called audit plan — which applies its transitions internally — sit ahead of the wrapper’s empty-diff exit, and the sync’s two halves are split across it (the audit transitions, E7 creates). A fully reverted PR has an empty diff — exactly the case where every open finding just got fixed — so gating the lifecycle behind the exit would strand resolved-in-practice threads open forever, blocking merges on repos that require conversation resolution.
• The summary comment still refreshes in place. gh_pr_comment (E8) takes its own marker, embeds it as a hidden tag, and find-and-edits the comment carrying it. Prefer a marker even for a lone plan: without one it falls back to --edit-last, which keys on the token identity, so the day a second plan comments on the same PR as the same bot the two summaries start overwriting each other. One marker per plan per comment surface.
• Inline comments stay advisory — but resolution now means something. They post as plain review comments, not a REQUEST_CHANGES review, so they only gate merging on repos with Require conversation resolution before merging enabled — and on those repos the lifecycle pulls its weight: fixed findings resolve themselves out of the gate’s way, while a finding the author disagrees with is one human “resolve conversation” click (read back as declined, never re-raised).

​

The workflow

• The release image is the runtime. container: ghcr.io/tylerdavis/graph:vX.Y.Z carries graph plus git, jq, and gh (pinned from official releases — distro packages lag the JSON fields and flags the pack tools use) and the safe.directory git config that Actions’ foreign-owned workspace mounts require. The image tag is the version pin; packages: read plus GITHUB_TOKEN credentials pull it while private.
• GRAPH_EVENTS=github makes a failing plan annotate the run itself — the one sanctioned case of graph writing non-deliverable output to stdout, failure-paths only (details).
• Guards are cheap and layered: drafts skipped, fork PRs skipped (they have no ANTHROPIC_API_KEY; if one slipped through, the unset ${VAR} makes the provider unusable and the first model call fails naming the variable — the backstop), one concurrency group per PR with cancel-in-progress.
• Cost per PR: worst case one judge-role call (docs-drift) plus two review-sized calls — the review itself, and the transition audit only when prior findings exist (each carries full file contents alongside the diff; the audit needs the same grounding to verify a fix). A first review pays for the review alone — the decide skips the audit outright; drafts pay nothing.
• One job per plan, no needs between them. The two jobs are independent and run in parallel; only docs-drift can fail the check (exit 4). The reviewer always exits 0 — its comments are the deliverable, not a gate.