| 1 | # GitHub App Setup (Codewhale Agent reviews) |
| 2 | |
| 3 | `codewhale review --pr N` writes an advisory code review of a pull request. With |
| 4 | `--post` (or from CI) the review is published to GitHub. Published reviews can |
| 5 | appear under two identities: |
| 6 | |
| 7 | - the default token the CI job already has (`github.token`), or |
| 8 | - a dedicated **GitHub App** so the review shows as a bot — e.g. |
| 9 | `codewhale-agent[bot]` — instead of a personal account. |
| 10 | |
| 11 | The App identity is optional. Nothing below is needed to run |
| 12 | `codewhale review --pr N` locally and print the report to your terminal. |
| 13 | |
| 14 | Related docs: |
| 15 | |
| 16 | - [Automatic Workflows](AUTOMATIC_WORKFLOWS.md) — the review workflow in context |
| 17 | - [Providers](PROVIDERS.md) — the model/key used to write the review |
| 18 | - [Receipts](RECEIPTS.md) — how posted reviews are anchored to a head SHA |
| 19 | |
| 20 | ## Account keys and provider keys |
| 21 | |
| 22 | `CODEWHALE_API_KEY` is a Codewhale account machine key (`cwc_key_…`), not a |
| 23 | vendor credential. In account mode the workflow first runs |
| 24 | `codewhale --no-project-config account agent` to check authentication and the |
| 25 | account's configured agent. It then selects the existing `codewhale` provider, |
| 26 | which sends that key to the Codewhale model relay. It never copies an account |
| 27 | key into a vendor environment variable. |
| 28 | |
| 29 | Connect the underlying provider in your Codewhale account and create a machine |
| 30 | key with `agent:run` and `models:infer` scopes. The default scopes from |
| 31 | `codewhale account api-keys create --name github-review` also include |
| 32 | `account:read`, which permits identity checks. Set `CODEWHALE_REVIEW_MODEL` to |
| 33 | an exact `provider/model` id returned by your account's authenticated |
| 34 | `GET /v1/models` catalog. The offline model defaults are bootstrap values, not |
| 35 | proof of account access; see [Providers](PROVIDERS.md). |
| 36 | |
| 37 | Bring your own provider key by setting its secret instead of |
| 38 | `CODEWHALE_API_KEY`: |
| 39 | |
| 40 | | Secret | Route | |
| 41 | | --- | --- | |
| 42 | | `CODEWHALE_API_KEY` | Codewhale account relay; requires an explicit account catalog model | |
| 43 | | `ZAI_API_KEY` | z.ai Coding Plan | |
| 44 | | `MODELSTUDIO_API_KEY` | Model Studio Token Plan | |
| 45 | | `DEEPSEEK_API_KEY` | DeepSeek | |
| 46 | | `OPENROUTER_API_KEY` | OpenRouter | |
| 47 | | `ANTHROPIC_API_KEY` | Anthropic | |
| 48 | |
| 49 | If both account and vendor secrets exist, the workflow selects account mode |
| 50 | and leaves every vendor variable unchanged. In this mode |
| 51 | `CODEWHALE_REVIEW_PROVIDER` must be unset or `codewhale`; a conflicting value |
| 52 | fails before review. To select BYOK, remove the account secret from this |
| 53 | workflow's configuration. |
| 54 | |
| 55 | ## Choosing the review route and model |
| 56 | |
| 57 | Configure repository variables under Settings → Secrets and variables → |
| 58 | Actions → Variables: |
| 59 | |
| 60 | | Variable | Account mode | BYOK mode | |
| 61 | | --- | --- | --- | |
| 62 | | `CODEWHALE_REVIEW_PROVIDER` | Unset or `codewhale` | Explicit provider, such as `deepseek` | |
| 63 | | `CODEWHALE_REVIEW_MODEL` | Required exact account catalog `provider/model` id | Optional exact model id; otherwise the provider's default | |
| 64 | |
| 65 | For BYOK without an explicit provider, the workflow chooses the first available |
| 66 | key in this order: z.ai, Model Studio Token Plan, DeepSeek, OpenRouter, |
| 67 | Anthropic. Set the provider explicitly when several keys are present. |
| 68 | |
| 69 | The workflow passes provider and model as global CLI flags before `review`, |
| 70 | with `--no-project-config`. Account mode deliberately pins the relay route: |
| 71 | the account agent precondition reports a configured provider, but does not |
| 72 | supply an exact model id or a vendor credential to the runner. |
| 73 | |
| 74 | For release PR **#6002** only, the workflow supplies an explicitly approved |
| 75 | `deepseek` / `deepseek-v4-pro` route and ceilings of **500000** characters per |
| 76 | pass, **16** complete passes, and **65536** output tokens per request. Existing |
| 77 | repository variables override these values. Other PRs retain the defaults |
| 78 | below. This exception changes no credentials or coverage rules: a diff that |
| 79 | requires more than 16 passes still fails before model review, and a provider |
| 80 | non-run is never completed-review evidence. Keep the release head frozen |
| 81 | during review to avoid cancellation and repeated provider cost. |
| 82 | |
| 83 | ## Complete diffs and input limits |
| 84 | |
| 85 | The workflow checks out the event's pinned head SHA for same-repository PRs, |
| 86 | and the pinned base SHA for fork PRs. It uses full history, fetches the base |
| 87 | repository's PR head ref, and verifies both event commits and a single merge |
| 88 | base. Fetching fork objects does not check out or execute their files, hooks, |
| 89 | submodules, or filters. Checkout credentials are not persisted. |
| 90 | [GitHub's checkout documentation](https://github.com/actions/checkout) describes |
| 91 | `fetch-depth: 0` and `persist-credentials: false`. |
| 92 | |
| 93 | The shared collector uses the complete GitHub diff when available and a |
| 94 | verified local Git diff when the API cannot provide it, including large PRs. |
| 95 | It rejects a changed snapshot, unavailable history, or incomplete diff before |
| 96 | review. Repository variable `CODEWHALE_REVIEW_MAX_CHARS` sets the input limit |
| 97 | per pass (default **200000**, allowed range **1–8388608**). The collector also has |
| 98 | an **8 MiB output** and **60-second command** bound; a character limit does not |
| 99 | bypass those transport bounds. |
| 100 | |
| 101 | A complete diff requiring more than one configured-limit pass fails the job by |
| 102 | default. It is never silently truncated or treated as a provider funding |
| 103 | problem. Repository variable `CODEWHALE_REVIEW_MAX_PASSES` (default **1**, |
| 104 | allowed range **1–64**) passes `--max-passes N` to the CLI. Raising it explicitly |
| 105 | authorizes the workflow to run up to N ordered passes for a complete review, |
| 106 | with additional provider cost and run time. Set it only after reviewing that |
| 107 | budget; leaving it unset retains one pass. If any pass fails, no partial |
| 108 | review is posted. |
| 109 | Increasing the character limit is a separate input-budget choice and still |
| 110 | requires a model with sufficient context. |
| 111 | |
| 112 | When the complete PR cannot fit the allowed pass count, keep the failed |
| 113 | advisory check and record that the model review did not run. Do not turn an |
| 114 | input-limit failure into a clean review. Maintainers can explicitly authorize |
| 115 | bounded whole-PR passes, or review bounded paths with a trusted build using |
| 116 | `review --base <base-sha> --path <path>` from a checkout pinned to the PR head. |
| 117 | Local diff reviews also reject oversized input. Path scopes cannot use `--pr` |
| 118 | or `--post`; their receipts cover only the selected paths. |
| 119 | Record the exact base/head, included paths and diff fingerprints, findings, |
| 120 | checks actually run, and remaining coverage. Separately review interactions |
| 121 | across paths and inspect changed media. A file inventory or a passing test |
| 122 | suite is not evidence that those source reviews completed. This fallback |
| 123 | does not change repository rules or satisfy a required whole-PR review. |
| 124 | |
| 125 | ## Review evidence and precision |
| 126 | |
| 127 | The Actions-backed GitHub App and the `review` tool use the same PR review |
| 128 | contract. Findings must explain an introduced defect's trigger, source evidence, |
| 129 | impact and a useful fix. Generic requests for more tests, style preferences and |
| 130 | unsupported compiler claims do not qualify as findings. An empty findings list |
| 131 | is valid; unresolved assumptions belong in the assessment. |
| 132 | |
| 133 | When the exact PR head is available locally, each pass also receives numbered |
| 134 | source excerpts around its changed hunks and nearby module declarations. These |
| 135 | come from regular Git blobs at the pinned head, never from dirty checkout files |
| 136 | or symlink targets. Source is not executed and no additional model call is made. |
| 137 | The excerpts use only the unused portion of `CODEWHALE_REVIEW_MAX_CHARS`, capped |
| 138 | at 50000 characters and 32 files per pass; individual blobs above 128 KiB are |
| 139 | omitted. The complete diff remains intact and remains the inline-comment scope. |
| 140 | |
| 141 | The request explicitly records unavailable files and omitted context. It does |
| 142 | not inspect unchanged caller files or run builds/tests, and a completed review |
| 143 | does not establish either. These source and local-fixture guarantees do not |
| 144 | establish a model's bug-detection rate or parity with another review product. |
| 145 | |
| 146 | ## Output budget |
| 147 | |
| 148 | `CODEWHALE_REVIEW_MAX_OUTPUT_TOKENS` optionally sets the CLI's output budget |
| 149 | through `CODEWHALE_MAX_OUTPUT_TOKENS`. Without it, the CLI chooses its automatic |
| 150 | cap. The workflow rejects values below **8192** to leave room for reasoning |
| 151 | and the final review. Provider accounting and supported limits vary; an empty |
| 152 | response is not proof of any one cause. A zero-exit review with empty output |
| 153 | fails the job. |
| 154 | |
| 155 | ## One-time setup, five steps |
| 156 | |
| 157 | You need owner access to the GitHub repository once. After setup, eligible non-draft |
| 158 | same-repository pull requests can post reviews as the App. |
| 159 | |
| 160 | 1. **Create the App.** GitHub → *Settings → Developer settings → GitHub Apps → |
| 161 | New GitHub App*. Name it (e.g. `Codewhale Agent`), set a homepage URL, and |
| 162 | **uncheck Webhook → Active** — the review is pulled on PR events by Actions, |
| 163 | so no webhook is needed. |
| 164 | 2. **Grant two repository permissions.** |
| 165 | - *Pull requests* → **Read & write** (to post the review and inline comments) |
| 166 | - *Contents* → **Read-only** (to read the diff; read-only is enough — avoid |
| 167 | write unless you have another reason) |
| 168 | Choose *Only on this account*, then **Create GitHub App**. |
| 169 | 3. **Download the private key.** On the App's page, *Private keys → Generate a |
| 170 | private key*. Keep the `.pem` file secret; it is the App's credential. |
| 171 | 4. **Install the App** on your account (*Install App* on the same page) and |
| 172 | select the repositories reviews should cover. |
| 173 | 5. **Add repository settings.** GitHub → *Settings → Secrets and |
| 174 | variables → Actions*: |
| 175 | |
| 176 | | Kind | Name | Value | |
| 177 | |----------|-----------------------------|-------------------------------------| |
| 178 | | Variable | `CODEWHALE_APP_ID` | the App ID shown on the App's page | |
| 179 | | Secret | `CODEWHALE_APP_PRIVATE_KEY` | the full `.pem` file contents | |
| 180 | | Secret | `CODEWHALE_API_KEY` | a Codewhale machine key; for BYOK use the provider's own secret name instead | |
| 181 | | Variable | `CODEWHALE_REVIEW_MODEL` | exact account catalog `provider/model` id (required for account mode) | |
| 182 | |
| 183 | App settings control identity. Model access separately requires a review |
| 184 | key and, for account mode, the catalog model. Optional budget variables are |
| 185 | `CODEWHALE_REVIEW_MAX_CHARS`, `CODEWHALE_REVIEW_MAX_PASSES`, and |
| 186 | `CODEWHALE_REVIEW_MAX_OUTPUT_TOKENS`. |
| 187 | |
| 188 | ## How the pieces connect |
| 189 | |
| 190 | [The review workflow](../.github/workflows/codewhale-review.yml) uses |
| 191 | `pull_request` for non-draft PRs targeting `main`. Only same-repository PRs |
| 192 | receive review secrets and build the candidate CLI. Fork PRs keep the trusted |
| 193 | base checkout and run only the diff-object checks; they receive no model or |
| 194 | App secrets and no model review. GitHub also |
| 195 | [withholds ordinary secrets from fork pull requests](https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows#workflows-in-forked-repositories). |
| 196 | Review a fork separately with a trusted build and deliberately provided |
| 197 | credentials. This workflow does not execute a fetched fork merely to obtain |
| 198 | a large diff. |
| 199 | |
| 200 | For eligible reviews, when |
| 201 | `CODEWHALE_APP_ID` **and** `CODEWHALE_APP_PRIVATE_KEY` are both present, the |
| 202 | job mints a short-lived installation token for the App |
| 203 | (`actions/create-github-app-token`) and hands it to the CLI as `GH_TOKEN`. |
| 204 | Otherwise it falls back to the workflow's own `github.token`. The CLI never |
| 205 | stores the token; each run mints a fresh one. |
| 206 | |
| 207 | The key-presence test lives in the job's `env:` block rather than its `if:` |
| 208 | because the `secrets` context is not available in a job-level `if:`. Job-level |
| 209 | `env` can read `secrets`, and step-level `if:` can read `env`, so build and review steps |
| 210 | gate on the non-secret string `env.HAS_ANY_KEY`. Diff preparation needs only |
| 211 | the workflow token with repository read access. Only booleans about presence |
| 212 | live at job scope; the key values are injected into the one step that runs the |
| 213 | review. |
| 214 | |
| 215 | Missing review credentials and provider HTTP failures keep the existing |
| 216 | advisory policy: the job can be green while the step summary explicitly says |
| 217 | **not run**. Provider failures also leave an idempotent non-run PR comment. |
| 218 | These are not clean-review results. Input-limit, snapshot, build, and other |
| 219 | review failures still fail the job. A successful later review removes a stale |
| 220 | non-run comment. |
| 221 | |
| 222 | The review itself is one **COMMENT** review — a summary body plus inline line |
| 223 | comments anchored to the PR head SHA. It never approves or requests changes; |
| 224 | CODEOWNERS stays the human authority. |
| 225 | |
| 226 | ## Running a review yourself |
| 227 | |
| 228 | ```sh |
| 229 | # print a report locally (uses your configured provider key) |
| 230 | codewhale review --pr 1234 |
| 231 | |
| 232 | # pin the route when a model is reachable through more than one provider |
| 233 | codewhale --provider deepseek --model MODEL_ID review --pr 1234 |
| 234 | |
| 235 | # account mode: check the agent, then use an exact id from the account catalog |
| 236 | codewhale --no-project-config account agent |
| 237 | codewhale --no-project-config --provider codewhale --model PROVIDER/MODEL_ID review --pr 1234 |
| 238 | |
| 239 | # explicitly increase a complete-diff input limit when needed |
| 240 | codewhale review --pr 1234 --repo OWNER/REPO --max-chars 6000000 |
| 241 | |
| 242 | # explicitly authorize at most 8 complete ordered model passes |
| 243 | codewhale review --pr 1234 --repo OWNER/REPO --max-passes 8 |
| 244 | |
| 245 | # publish it to GitHub as whichever identity GH_TOKEN carries |
| 246 | codewhale review --pr 1234 --post |
| 247 | ``` |
| 248 | |
| 249 | `GH_TOKEN` may be your `gh` CLI token (posts as you) or an App installation |
| 250 | token (posts as the App). The `--post` flag is always opt-in. |
| 251 | |
| 252 | ## Troubleshooting |
| 253 | |
| 254 | - **Review posts as you, not the bot.** The variable or the private-key secret |
| 255 | is missing/empty; the job silently falls back to `github.token`. Check both |
| 256 | names character-for-character. |
| 257 | - **Step summary says "not run".** No model review completed. Check whether |
| 258 | this is a fork, review credentials are missing, or the provider failed. |
| 259 | - **Account model or provider error.** Set the provider to `codewhale` (or |
| 260 | unset it), choose the exact model from the account catalog, and check that |
| 261 | the machine key has the required scopes and the account has a configured |
| 262 | agent. A vendor key belongs in its own secret, never `CODEWHALE_API_KEY`. |
| 263 | - **Complete diff exceeds the input limit.** Inspect the reported size and |
| 264 | model context capacity before raising `CODEWHALE_REVIEW_MAX_CHARS`. An 8 MiB |
| 265 | transport-bound failure cannot be bypassed with that variable. |
| 266 | - **PR head changed or history is unavailable.** Rerun for the current |
| 267 | revision. The workflow refuses to review an unverified snapshot. |
| 268 | - **"available from configured provider route(s): ...".** Two provider keys are |
| 269 | configured and the model is reachable from both. Set repository variable |
| 270 | `CODEWHALE_REVIEW_PROVIDER`. |
| 271 | - **Empty review.** The job fails. Inspect provider errors and output-budget |
| 272 | receipts; increasing `CODEWHALE_REVIEW_MAX_OUTPUT_TOKENS` may help when |
| 273 | reasoning exhausted the budget, but does not diagnose the cause by itself. |
| 274 | - **App token step fails.** The `.pem` was regenerated after the secret was |
| 275 | set — paste the newest key into `CODEWHALE_APP_PRIVATE_KEY` again, and |
| 276 | confirm the App is actually installed on the repository. |
| 277 | - **Name already taken.** GitHub App names are global; pick another name. The |
| 278 | bot's display login is `<slug>[bot]`, derived from the name. |
| 279 |