---
title: izri/tests
description: Are the tests for this change any good? Do they pass, do they run the changed lines, and do they check anything.
order: 30
---

# `izri/tests`

Tests answers one question about a pull request: **are the tests for this change any good?** That breaks into three:

1. **Do they pass?**
2. **Do they run the lines this change touched?**
3. **Do the tests this PR wrote check anything?**

Izri judges the delta a PR introduces. The rest of your project is context for that judgement — it is never what's being graded. So coverage here always means *diff* coverage: of the lines this PR added or modified, how many did a test execute. Project-wide coverage is your coverage tool's job.

## 1. Pass / fail

A failed run fails `izri/quality`, whatever the other signals say. One red test must never read as "safe to merge".

| Finding | Meaning |
| --- | --- |
| `test_failure` | The run finished with failing tests. **Hard rule.** The failed tests are listed on the report. |
| `runner_error` | The run broke before producing results. **Hard rule** — no evidence never resolves to "safe". |
| `zero_tests` | A passing run executed no tests. **Hard rule, opt-in** via `block.zero_tests`. |
| `evidence_missing` | The project uploads from CI, and nothing arrived for this commit within the grace window. **Hard rule.** |
| `test_evidence_untrusted` | Green evidence from a PR that edited what produced it. `warn`; **hard, opt-in** via `block.untrusted_evidence`. |

### Run statuses

| Status | Meaning |
| --- | --- |
| `PENDING` | Queued, not yet picked up. |
| `RUNNING` | Executing. |
| `PASSED` | Terminal. Everything passed. |
| `FAILED` | Terminal. Tests failed. |
| `ERROR` | Terminal. The run itself broke. |
| `CANCELLED` | Terminal. Stopped before completion. |

`PASSED` is the only terminal status that isn't a failure. The CLI exits non-zero on every other one.

## 2. Diff coverage

A green suite says nothing about whether it touched your change. Here is the case this exists for.

An agent is asked to fix a rounding bug in `calculateTax`. It changes three lines and adds `tax.test.ts` with four new tests. CI is green.

```ts
// src/billing/tax.ts — the change
export function calculateTax(cents: number, rate: number): number {
-  return Math.round(cents * rate)
+  return Math.floor(cents * rate + 0.5)
}
```

```ts
// tests/tax.test.ts — the new tests
import { formatTax } from '../src/billing/format'

it('formats a tax amount', () => {
  expect(formatTax(1050)).toBe('$10.50')   // never calls calculateTax
})
```

Four passing tests, none of which call `calculateTax`. Project coverage barely moves. Izri reports `change_uncovered` and the verdict fails: a suite that grows without touching the diff manufactures confidence.

Each changed file lands in one bucket — `covered`, `partial`, `uncovered`, or `notInstrumented` (the coverage tool never saw the file: a deliberate exclusion, or a real gap; Izri says so at `info` rather than guessing). Uncovered lines are annotated inline on the pull request.

| Finding | Severity | Meaning |
| --- | --- | --- |
| `change_uncovered` | `error` | A changed file where no test executed any changed line. **Hard rule.** |
| `change_partially_covered` | `warn` | Some changed lines were executed, some weren't. |
| `low_diff_coverage` | `warn` | Under 60% of the added lines were executed. Named `low_overall_coverage` before September 2026. |
| `file_not_instrumented` | `info` | The coverage tool never saw the file. |
| `diff_coverage_below` | `error` | Diff coverage is under your project's minimum. **Hard rule, opt-in.** |

### Coverage minimum

Partial coverage doesn't block by default — real code often has a branch the tests don't reach. To require more, set a minimum in the project's merge policy:

```yaml
policy:
  block:
    diff_coverage_below: 80
```

A delta whose diff coverage is under 80% then fails with `diff_coverage_below`. It is off by default, it never fires when coverage wasn't measured, and it stays quiet when `change_uncovered` already fired for the same delta — one fact, one blocking finding.

## 3. Test quality

Findings about the tests this PR wrote. **They are advisory: shown on the report, never scored, never blocking.** Heuristics about someone's assertions are opinionated, and a false "your tests are weak" that blocks a merge is how a gate gets switched off. A category is promoted only after it has measured precision on real pull requests.

| Finding | Severity | Meaning |
| --- | --- | --- |
| `change_lacks_test` | `warn` | A changed source file has no changed test importing it. Static; a nudge, not a verdict. |
| `weak_assertion` | `warn` | An LLM judged the assertions too weak to catch a regression. Listed separately as LLM-derived. |
| `assertion_free_test` | `warn` | A test block this PR touched contains no assertion. |
| `tautological_assertion` | `warn` | An assertion that cannot fail, such as `expect(true).toBe(true)`. |
| `test_mocks_subject` | `warn` | A test mocks a module this PR changed and no test file in the PR imports that module unmocked, so the change may never run under test. Mocking a changed collaborator is fine when its own test is in the PR. |
| `test_passes_without_change` | `warn` | A test this PR wrote passes with the change reverted. Found by [the revert check](/docs/signals/hallucination#the-revert-check). |

## Scoring

The tests score is the **lower of pass rate and diff coverage**, 0–100, and carries weight **0.36** in the composite by default. Green tests that ran half of the change score 50, not 100. With no coverage measured it is the pass rate alone.

## Where the evidence comes from

Most teams already run their tests in CI, so Izri doesn't run them again. Each project has one **evidence source**, set in the dashboard or with `tests.evidence` in [`.izri/config.yml`](/docs/configure/config-yml):

| Source | What happens | When to use it |
| --- | --- | --- |
| `ci_upload` | Your CI runs the tests and uploads the JUnit results, plus a coverage report, with `izri upload`. Izri judges that run. | Any framework, any CI. The default once a project has made a trusted upload. |
| `runner` | Izri's runner executes the tests relevant to the diff and collects coverage itself. Playwright only. | Projects with no CI of their own. |
| `none` | No tests verdict. The row shows as `skipped` and its weight goes to the other signals. | New projects until they pick one; the first trusted upload switches to `ci_upload`. |

### Uploading from CI

```yaml
jobs:
  tests:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      id-token: write # lets Izri trust the upload
    steps:
      - uses: actions/checkout@v4
      - run: npx vitest run --reporter=junit --outputFile=junit.xml --coverage --coverage.reporter=lcov
        continue-on-error: true
      - uses: legendify-dev/izri/apps/github-action@v1
        with:
          api-token: ${{ secrets.IZRI_API_TOKEN }}
          mode: upload
          junit-file: junit.xml
          coverage-file: coverage/lcov.info
```

or `izri upload --junit junit.xml --coverage coverage/lcov.info` from any CI. LCOV and Cobertura coverage are accepted. **Always upload, even when nothing ran:** `izri upload --skipped "docs-only change"` records an explicit empty run. Silence is treated as a broken step, not as a clean one.

### When the upload doesn't arrive

A project on `ci_upload` whose CI never uploads for a commit holds `izri/quality` at `unknown` for a grace window (six hours by default), then fails with `evidence_missing`: no evidence never resolves to "safe", and a failing check with a cause beats a pending one without.

### Evidence the PR could have shaped

A pull request that edits the workflow that produced the evidence, a composite action, a test-runner or build config (`vitest.config.*`, `jest.config.*`, `playwright.config.*`, `tsconfig*.json`, `babel.config.*`, `pytest.ini`, `pyproject.toml`, …), a test setup file, a `Makefile`, the test scripts in `package.json`, or `.izri/` could make its own tests look green. The list is a heuristic and cannot be complete; it catches the common ways, not every way. Izri checks with GitHub — never with the upload — what the PR changed.

Which workflow produced the evidence comes from the GitHub Actions id-token sent with the upload in the `X-Izri-OIDC` header. The job needs `permissions: id-token: write`; `izri upload` and the Action request and send the token themselves, and the [REST reference](/docs/reference/rest-api) shows how to send it when calling the route directly. Its signed claims name the repository, the commit, the run and the workflow file, and for a reusable workflow the file whose job ran. Only those workflow files taint when edited; editing any other workflow changes nothing.

Green evidence is trusted only when all of this holds:

- the upload carries a valid id-token for the project's repository — without one (a local `izri upload`, another CI system) it is stored and judged but never trusted, because anyone holding a project token could have sent it for any commit;
- GitHub confirms the run is of that workflow on this commit;
- a GitHub App installation on the repository could list what the PR changed, and the PR changes fewer than 300 files (past that GitHub cannot list them all, so they cannot be vetted);
- the PR edits nothing on the list above.

Red stays red whoever produced it. Untrusted green is still scored but carries a `test_evidence_untrusted` warning giving the reason; the opt-in `block.untrusted_evidence` policy rule makes it block instead.

On the delta report, the `izri/tests` card and its test-run dialog show who produced uploaded evidence: *Produced by `<workflow file>` (run #n)*, linking to the GitHub Actions run, with the job's workflow added when a reusable workflow ran it, and the actor and event in the dialog. Green evidence that is not trusted carries the reason: in the dialog, and on the card unless its `test_evidence_untrusted` finding already says it. A red run, or an upload the verdict did not use, names its producer and nothing about trust.

### The runner

With `runner`, runs execute on Izri's runner — the queue consumer that drains jobs from Redis — so a self-hosted instance without a runner leaves runs at `PENDING` indefinitely. See [self-hosting](/docs/install/self-hosting). `izri run --watch` triggers one and polls until it is terminal; the GitHub Action does the same with `mode: tests`. A project on another source still gets a runner job when `izri/visual` needs screenshots; that job is labelled "visual capture" and is not the tests verdict.

## What to do when it fails

- **Red tests:** fix them; the report names each one.
- **`change_uncovered` or a coverage minimum:** write a test that calls the changed code. Not more tests — the *right* test. The inline annotations show exactly which lines went unexercised.

## Related

- [`izri/hallucination`](/docs/signals/hallucination) — whether what the change says about itself is true.
- [Verdicts and exit codes](/docs/reference/verdicts-and-exit-codes).
- [Troubleshooting](/docs/reference/troubleshooting) — stuck at `PENDING`.