Your CI logs are not evidence

The first time somebody outside your team asks you to demonstrate that a release was checked before it shipped, the instinct is to send a screenshot of a green pipeline.

It is a reasonable instinct and it does not survive contact. Not because anyone doubts you, but because a green build answers a question nobody asked.

The follow-up questions#

A screenshot says a job succeeded. Here is what it does not say, and what will be asked:

• Which version? The pipeline ran on a branch. Branches move. Naming a branch describes something that has since changed, and the artifact you shipped may not be what was checked.
• Which checks ran? A green build with five passing steps and one that was skipped by a path filter looks exactly like a green build with six.
• Did anything fail to start? A scanner whose binary was missing, a step with continue-on-error, a job that timed out and was retried into success, all green.
• Against what data? Vulnerability feeds have dates. The same code scanned two weeks apart gives different answers, and only one of those runs knows it was working from stale data.
• What was decided? A clean report showing zero findings because three were suppressed is a different document from one showing zero because none were found. Both look clean.
• Would it produce the same answer twice? If not, the run describes one event rather than establishing a property.

Six questions, and a screenshot answers none of them. Worse, CI logs usually expire, thirty or ninety days on most platforms, so the thing you pointed at will not exist when it is asked for.

The distinction that matters#

A claim is an assertion that something is true. Evidence is an artifact that stands on its own, later, without you in the room to explain it.

A green tick is a claim. It is entirely credible, and it is still a claim, because everything that would let a stranger check it lives in your memory and in logs that are about to be deleted.

The test is simple: hand it to somebody who does not trust you and is not hostile, a new team member, an auditor, your future self after eighteen months. Can they tell what was checked and what it concluded? If they have to ask you, it is not evidence yet.

What an artifact needs to carry#

Not much, and almost none of it is hard once you decide to keep it:

• The resolved revision, a commit, not a branch name.
• The controls that ran, and the ones that did not, with their verdicts. Explicitly.
• Versions of the tools, because two reports from different scanner versions are not comparable and a reader comparing quarters will assume they are.
• The dates of the data it was judged against.
• Every suppression, with its reason and who accepted it. See risk acceptance.
• A timestamp, and a format somebody else can read. SARIF exists for this and is understood by tooling you do not control.

Keep that per release, somewhere with a retention policy longer than your CI's, and the six questions above answer themselves.

The failure that costs most#

A check that did not run, reported as a pass.

It is the most common defect in hand-assembled pipelines and the hardest to detect afterwards, because nothing in the output distinguishes it from a genuine pass. The build is green, the report is clean, and the missing control is discovered when somebody goes looking for a finding that should have been there.

Any artifact worth calling evidence reports a control that could not run as an error, never as an absence. That is a property to check your tooling for, and it is worth checking directly: delete a scanner's binary and see what your pipeline says. If it still passes, your evidence has been lying by omission for as long as you have had it.

The short version#

Reproducibility is what turns a result into evidence. If the same input would not give the same answer, because the analysis is non-deterministic, or the ordering shifts, or it depends on what a model said that day, then you have an anecdote with a timestamp.

That is a good enough reason to keep detection and gating deterministic, quite apart from the engineering arguments for it. The people who ask for evidence are not the people you build for day to day, and they will only ever see the artifact.

The audit evidence guide covers this in full, and compliance frameworks covers who is asking and why.