Your CI logs are not evidence
On this page
The first time somebody outside your team asks you to demonstrate that a release was checked before it shipped, the instinct is to send a screenshot of a green pipeline.
It is a reasonable instinct and it does not survive contact. Not because anyone doubts you, but because a green build answers a question nobody asked.
The follow-up questions
A screenshot says a job succeeded. Here is what it does not say, and what will be asked:
| The question | Why the screenshot cannot answer it |
|---|---|
| Which version? | the pipeline ran on a branch, and branches move. What you shipped may not be what was checked |
| Which checks ran? | five passing steps and one skipped by a path filter looks exactly like six |
| Did anything fail to start? | a missing binary, a continue-on-error, a timeout retried into success. All green |
| Against what data? | feeds have dates, so the same code two weeks apart gives different answers, and only one run knows it was stale |
| What was decided? | zero findings because three were suppressed is a different document from zero because none were found |
| Would it answer the same twice? | if not, the run describes an event rather than establishing a property |
Six questions, and a screenshot answers none of them. Worse, CI logs usually expire, thirty or ninety days on most platforms, so the thing you pointed at will not exist when it is asked for.
The distinction that matters
A claim
An assertion that something is true. A green tick is one, and it is entirely credible, which is not the problem. Everything that would let a stranger check it lives in your memory and in logs about to be deleted.
Evidence
An artifact that stands on its own, later, without you in the room to explain it. It names what ran, what did not, what it was judged against, and what was decided.
The test is simple: hand it to somebody who does not trust you and is not hostile, a new team member, an auditor, your future self after eighteen months. Can they tell what was checked and what it concluded? If they have to ask you, it is not evidence yet.
What an artifact needs to carry
Not much, and almost none of it is hard once you decide to keep it:
| What it carries | What its absence costs a reader |
|---|---|
| the resolved revision, a commit rather than a branch name | a branch has moved since, so they cannot tell what was checked |
| the controls that ran, and the ones that did not, with verdicts | they cannot tell coverage from silence |
| versions of the tools | two reports from different scanner versions are not comparable, and somebody comparing quarters will assume they are |
| the dates of the data it was judged against | they cannot tell a clean result from a stale feed |
| every suppression, with its reason and who accepted it | zero findings reads as nothing found. See risk acceptance |
| a timestamp, in a format somebody else can read | they need their own tooling to open it. SARIF is understood by tools you do not control |
Keep that per release, somewhere with a retention policy longer than your CI's, and the six questions above answer themselves.
The failure that costs most
A check that did not run, reported as a pass.
It is the most common defect in hand-assembled pipelines and the hardest to detect afterwards, because nothing in the output distinguishes it from a genuine pass. The build is green, the report is clean, and the missing control is discovered when somebody goes looking for a finding that should have been there.
Any artifact worth calling evidence reports a control that could not run as an error, never as an absence:
CONTROLS
iac FAIL 4 P1 5 P2 15 P3 16 P4
licenses FAIL 353 P2 178 P3
sast FAIL 7 P1 8 P2 2 P4
sca FAIL 18 P1 18 P2 2 P3
dast ERROR nuclei not installed
draugr tools install nuclei installs what dast needsERROR is its own state, beside pass and FAIL rather than folded into either. The row says
which tool was missing, and the line under the controls is the command that installs it. A reader a year from now can see that the dynamic check was
not performed, which is the one thing an absence can never tell them.
That is a property to check your own tooling for, and it is worth checking directly: delete a scanner's binary and see what your pipeline says. If it still passes, your evidence has been lying by omission for as long as you have had it.
The short version
Reproducibility is what turns a result into evidence. If the same input would not give the same answer, because the analysis is non-deterministic, or the ordering shifts, or it depends on what a model said that day, then you have an anecdote with a timestamp.
That is a good enough reason to keep detection and gating deterministic, quite apart from the engineering arguments for it. The people who ask for evidence are not the people you build for day to day, and they will only ever see the artifact.
The audit evidence guide covers this in full, and compliance frameworks covers who is asking and why.