The ZAP Baseline: Zero Failures Is Not the Number That Matters

The owner's request landed before the QA backfill wave began: run the ZAP baseline again, because a new surface had appeared since the previous day's fixing session, user authentication through Google OAuth. The result was read straight from the zap-baseline.json artifact, not from the console transcript. The numbers: zero FAIL, four WARN, 57 PASS. Triggered by a new third-party authorization layer, the initial expectation ran the other way: integrations like that usually arrive with loose headers or leaking content policies, and at least one new alert seemed certain.

The delta table, not the single number

What makes this session trustworthy is not the zero. It is the rule-level comparison against the baseline from the day before. Every line moved down or stayed flat: CSP wildcard directive from three instances to two, script-src unsafe-eval three to two, script-src unsafe-inline three to two, style-src unsafe-inline three to two, timestamp disclosure two to one, suspicious comments eleven to nine, and modern web application flat at five. No new rules, no rising instances. The OAuth surface nobody trusted produced not a single passive-level alert. Those numbers were not hand-counted: the session report requires verification from the JSON file, and recounting the instances per rule from the artifact matched the table. The machine that produced the evidence also closed the question.

One reading correction belongs in the record. The report headline says four warnings, while the JSON artifact holds seven alert instances: four Medium, one Low, two Informational. The four counts the Medium instances configured as WARN; the rest register as PASS or informational. The two readings complement each other, and quoting either one alone would leave report readers with the wrong picture of the application.

Why passive is enough for a tripwire

The official baseline script spiders the target for one minute by default, waits for the passive scan to finish, then reports without ever attacking [1]. All alerts come out as WARN by default, and a configuration file can promote specific rules to FAIL or ignore them [1]. For a fast regression gate before a backfill, that behavior fits: cheap, safe to repeat, and sensitive enough to notice header changes, cookie flags, or information leaks. Passive mode also means one session changes nothing on the target: no form is submitted, no test data is left behind, so a baseline can run at any moment without special coordination. That low cost is what makes comparing between sessions a habit, and the delta table only means something if comparing is a habit.

The limits are written into the session report too. The target was a development server over plain HTTP without a reverse proxy, and the crawl ran unauthenticated, so the admin area was never touched. Formal session-management testing in the style of the OWASP Web Security Testing Guide requires a legitimate session context, something this passive scan never had [2]. The guide positions itself as the reference framework for web application security testing, and this baseline is its thinnest slice: enough to catch configuration regressions, still short of session logic [2].

That limitation has one practical consequence: the replay session for the admin and API surface runs separately, and until it is repeated, claims of safety apply only to the public pages reachable without login. Mixing the two scopes into one concluding sentence is the fastest way to make a security report misleading.

Zero FAIL is still good news. The durable lesson is how the number was earned: compare rule-level deltas between sessions, read from machine artifacts, and record scope limits upfront. Without all three, a zero is easily mistaken for a guarantee, when it only means safe on the layer actually scanned.

Sources: [1] official zap-baseline.py documentation [2] OWASP Web Security Testing Guide