Audit evidence: what makes a scan report count

In plain terms#

At some point somebody outside your team, an auditor, a customer's security reviewer, a procurement questionnaire, will ask you to demonstrate that your software is checked before it ships. Not to assert it. To demonstrate it.

That is a different artifact from the one your team uses day to day. A dashboard is for you. An auditor needs something that stands on its own, months later, without you in the room to explain it.

Why it matters#

The instinct is to send a screenshot of a green build, and it fails for a reason worth understanding: a screenshot is a claim, not evidence. It shows a result and nothing about what produced it. Was that this release or last week's? Which checks ran? Did one of them error and get counted as a pass? Would running it again produce the same answer?

Evidence answers those without being asked. That is the whole difference.

What a report needs to carry#

• What was checked. Not "the repository", the exact revision. A report that names a branch describes something that has since changed.
• What was checked for. Which controls ran, and which did not. A report listing five passing checks, where a sixth failed to start, is worse than one listing six honestly.
• Against what. Vulnerability data has a date. A scan run against a two-week-old feed and one run this morning give different answers about the same code, and only one of them says so.
• What was decided. Findings that were accepted, by whom, and why. See risk acceptance. A report showing zero findings because three were suppressed is not the same document as one showing zero because none were found.
• When, and by what. The tool, its version, the time. Two reports from different versions of a scanner are not comparable, and an auditor comparing quarters will assume they are.

Reproducibility is the property that makes it evidence#

The question underneath all of this is: would this produce the same answer twice?

If the answer is no, because the analysis is non-deterministic, or the ordering shifts, or the result depends on what a model happened to say that day, then the report describes one event rather than establishing a property. That may still be useful. It is not evidence.

This is why deterministic detection matters more in a compliance context than it does in a purely security one. A security team can live with a tool that finds slightly different things each run. An audit trail cannot, because its whole function is to be checkable by somebody who was not there.

The failure that costs the most#

A check that did not run, reported as a pass.

It is the most common defect in home-grown pipelines and the hardest to see afterwards: a scanner's binary was missing, the step had continue-on-error, the job was skipped by a path filter. The build is green. The report is clean. Nothing in the artifact distinguishes it from a genuine pass, and it will not be noticed until somebody goes looking for a finding that should have been there.

Any evidence artifact worth the name reports a control that could not run as an error, not as an absence.

How Draugr fits#

Draugr produces the artifact rather than a dashboard: a verdict, plus SARIF and a JSON report written to a directory you keep.

Each report names the repository and the resolved revision it scanned, the controls that ran and their verdicts, the scanner versions, the dates of the exploitability feeds used, and every suppression with its reason and the person who accepted it. A control whose scanner is missing is an error, never a silent pass. That is one of the project's stated design rules rather than a feature.

Detection, severity and the gate are deterministic by design: the same input produces the same verdict, so two reports from different quarters can be compared and a re-run can be demanded.

Which framework you are being audited against, and which of its clauses a finding speaks to, stays a judgement about your obligations. What Draugr gives the people holding it is the artifact they need to answer with: a dated, reproducible record of what was checked, what it concluded, and who decided about the rest.