Audit evidence: what makes a scan report count
On this page
In plain terms
At some point somebody outside your team, an auditor, a customer's security reviewer, a procurement questionnaire, will ask you to demonstrate that your software is checked before it ships. Not to assert it. To demonstrate it.
That is a different artifact from the one your team uses day to day. A dashboard is for you. An auditor needs something that stands on its own, months later, without you in the room to explain it.
Why it matters
The instinct is to send a screenshot of a green build, and it fails for a reason worth understanding: a screenshot is a claim, not evidence. It shows a result and nothing about what produced it. Was that this release or last week's? Which checks ran? Did one of them error and get counted as a pass? Would running it again produce the same answer?
Evidence answers those without being asked. That is the whole difference.
What a report needs to carry
| The question | Why a report that skips it fails |
|---|---|
| What was checked | not "the repository" but the exact revision. A report naming a branch describes something that has since changed |
| What was checked for | which controls ran, and which did not. Five passing checks where a sixth failed to start is worse than six reported honestly |
| Against what | vulnerability data has a date. A scan against a two-week-old feed and one run this morning give different answers about the same code, and only one says so |
| What was decided | findings accepted, by whom, and why. Zero findings because three were suppressed is not the same document as zero because none were found. See risk acceptance |
| When, and by what | the tool, its version, the time. Two reports from different scanner versions are not comparable, and an auditor comparing quarters will assume they are |
Reproducibility is the property that makes it evidence
The question underneath all of this is: would this produce the same answer twice?
If the answer is no, because the analysis is non-deterministic, or the ordering shifts, or the result depends on what a model happened to say that day, then the report describes one event rather than establishing a property. That may still be useful. It is not evidence.
This is why deterministic detection matters more in a compliance context than it does in a purely security one. A security team can live with a tool that finds slightly different things each run. An audit trail cannot, because its whole function is to be checkable by somebody who was not there.
The failure that costs the most
A check that did not run, reported as a pass. It is the most common defect in home-grown pipelines and the hardest to see afterwards: a scanner's binary was missing, the step had continue-on-error, the job was skipped by a path filter. The build is green. The report is clean. Nothing in the artifact distinguishes it from a genuine pass, and it will not be noticed until somebody goes looking for a finding that should have been there.
Any evidence artifact worth the name reports a control that could not run as an error, not as an absence.
How Draugr fits
The artifact, not a dashboard
Draugr produces the artifact rather than a dashboard: a verdict, plus SARIF and a JSON report written to a directory you keep.
It names what it read
Each report names the repository and the resolved revision it scanned, the controls that ran and their verdicts, the scanner versions, the dates of the exploitability feeds used, and every suppression with its reason, and for each exclusion the commit, author and date its git history records. A control whose scanner is missing is an error, never a silent pass. That is one of the project's stated design rules rather than a feature.
The same input, the same answer
Detection, severity and the gate are deterministic by design: the same input produces the same verdict, so two reports from different quarters can be compared and a re-run can be demanded.
Which framework you are being audited against, and which of its clauses a finding speaks to, stays a judgement about your obligations. What Draugr gives the people holding it is the artifact they need to answer with: a dated, reproducible record of what was checked, what it concluded, and who decided about the rest.
Reports and publishers has the formats a run can write and where each can be sent, which is the practical question behind keeping any of this.