Read this tutorial on Signadot docs.

Static scanners read code. Some security bugs only show up when the code runs: a query that turns injectable once real input reaches it, or an error page that leaks database details. Catching those takes a running target, and in a microservices system that usually means a shared staging environment, after the change is merged, where a finding could come from any of the changes deployed together.

This tutorial runs ZAP, the most widely used open source dynamic application security testing (DAST) scanner, against a single change before merge. The changed service runs in a Signadot sandbox, and everything else it talks to comes from the cluster's baseline. ZAP runs in the cluster as a Signadot Job, which adds the sandbox's routing key to every request ZAP sends, so its attacks reach your change and nothing else.

You'll make a change to HotROD's location service that adds search by name, and introduces a SQL injection along with it. ZAP catches it. You fix it and scan again.

ZAP's requests carry the sandbox's routing key, so they reach the location service with your change, running on your machine. Requests without the key, like the frontend's, reach the baseline location service. Both versions use the cluster's MySQL.

The changed service runs on your machine, so you don't build or push an image, and a fix is one restart away from its next scan. The same Job also scans a change deployed in the cluster, which is how it runs as a pull request check.

Estimated time: 20 minutes.

What ZAP detects

ZAP's rules come in two kinds. Passive rules only inspect the responses to ordinary requests, so they're safe to run against any sandbox. Active rules send attack payloads and watch how the application reacts.

Category Examples Rules
Injection SQL injection, OS command injection, server-side template injection, XML external entities, CRLF injection Active
Cross-site scripting Reflected and stored XSS Active
File access Path traversal, remote file inclusion Active
Information disclosure Database errors, stack traces and debug messages in responses, server version numbers in headers Passive
Missing security headers Content Security Policy, anti-clickjacking, X-Content-Type-Options, Strict-Transport-Security Passive
Weak cookie settings Cookies without the HttpOnly, Secure or SameSite attributes Passive
Cross-origin misconfiguration Overly permissive CORS headers Passive

Each finding has a risk level (High, Medium, Low or Informational), the request that triggered it, and a suggested fix. ZAP's Docker image ships three packaged scans built on these rules:

Scan What it does Use it for
Baseline (zap-baseline.py) Crawls the target and runs passive rules only Web UIs, and any sandbox whose data isn't isolated
API (zap-api-scan.py) Imports an OpenAPI, SOAP or GraphQL definition and runs active rules against each operation APIs. It skips browser-only checks such as XSS
Full (zap-full-scan.py) Crawls the target and runs all active rules Web apps, when the sandbox's data is isolated

This tutorial uses the API scan.

Before you begin

You need:

  • A Kubernetes cluster with the Signadot Operator installed.
  • The Signadot CLI, logged in with signadot auth login. Creating a job runner group requires a Signadot admin.
  • kubectl pointed at that cluster, plus git, jq and Go 1.21 or later. Go downloads the toolchain version HotROD asks for.

1. Install HotROD

Skip this step if HotROD already runs in the hotrod namespace. The files in this tutorial assume that namespace.

kubectl create namespace hotrod kubectl -n hotrod apply -k 'https://github.com/signadot/hotrod/k8s/overlays/prod/devmesh' kubectl -n hotrod wait --for=condition=available deployment --all --timeout=5m 

2. Get the change

The change is on HotROD's zap-location-search branch. Clone that branch and the examples repository side by side, then copy the Signadot files into HotROD:

git clone https://github.com/signadot/examples.git signadot-examples git clone -b zap-location-search https://github.com/signadot/hotrod.git cd hotrod cp -R ../signadot-examples/zap-security-tutorial/security .signadot/ 

The change adds an optional name filter to GET /locations. Here's the core of the commit, in services/location/database.go:

query := "SELECT id, name, coordinates FROM locations" if name != "" { query += fmt.Sprintf(" WHERE name LIKE '%%%s%%'", name) } rows, err := d.db.Query(query) 

It returns the right rows for normal input, so it could pass a quick review.

The files you copied into .signadot/security/:

File What it does
sandbox.yaml A sandbox that routes requests for the location service to port 8081 on your machine
zap-runner.yaml A job runner group that runs Jobs in ZAP's official image
zap-api-scan.yaml The scan: a ZAP API scan of GET /locations, sent with the sandbox's routing key

3. Run the change in a sandbox

sandbox.yaml is a local sandbox: requests that carry its routing key and are bound for the location service go to port 8081 on your machine.

name: location-search spec: cluster: "@{cluster}" description: location service with name search, running on my machine local: - name: location from: kind: Deployment namespace: hotrod name: location mappings: - port: 8081 toLocal: localhost:8081 

Connect your machine to the cluster. signadot local connect asks for your password, because it updates /etc/hosts and local networking:

signadot local connect 

Create the sandbox. Set CLUSTER to your cluster's name from signadot cluster list:

export CLUSTER=my-cluster signadot sandbox apply -f .signadot/security/sandbox.yaml --set cluster=$CLUSTER 

In a second terminal, in the hotrod directory, start the location service with your change. get-env loads the baseline's environment, so the service connects to the cluster's MySQL and Redis:

eval "$(signadot sandbox get-env location-search)" go run ./cmd/hotrod location 

Leave it running. Back in the first terminal, send the same request twice: once as a normal request, and once with the sandbox's routing key:

ROUTING_KEY=$(signadot sandbox get location-search -o json | jq -r .routingKey) curl "http://location.hotrod:8081/locations?name=Coffee" curl -H "baggage: sd-routing-key=$ROUTING_KEY" "http://location.hotrod:8081/locations?name=Coffee" 

The first request reaches the baseline, which ignores name and returns all five locations. The second reaches the code on your machine and returns only Amazing Coffee Roasters. Both requests go to the same address. The routing key decides which version handles them.

4. Create the ZAP job runner group

A job runner group keeps a pod running ZAP's image in your cluster, ready to run scan Jobs. zap-runner.yaml:

name: zap spec: cluster: "@{cluster}" namespace: signadot-tests image: ghcr.io/zaproxy/zaproxy:stable jobTimeout: 15m scaling: manual: desiredPods: 1 

Create its namespace, then the runner group:

kubectl create namespace signadot-tests signadot jobrunnergroup apply -f .signadot/security/zap-runner.yaml --set cluster=$CLUSTER 

The first time, the pod pulls ZAP's image, which takes a minute or two. Run signadot jobrunnergroup get zap until it shows Status: 1/1 pods ready.

5. Scan the sandbox

zap-api-scan.yaml is the scan Job:

spec: namePrefix: zap-api-scan runnerGroup: zap # Ties the Job to the sandbox, and re-queues it if the sandbox isn't ready.  routingContext: sandbox: "@{sandbox}" # Adds the sandbox's routing key to every plain HTTP request the Job sends.  # ZAP targets the baseline service address, and Signadot routes its requests  # to the sandboxed workload.  trafficManager: injectRoutingKey: auto script: | #!/bin/bash  set -euo pipefail mkdir -p /zap/wrk && cd /zap/wrk # Scan only what the change touches: GET /locations and its new  # `name` parameter. ZAP sends attack payloads to every operation listed  # here, so write operations are left out.  cat > openapi.yaml <<'EOF' openapi: 3.0.3 info: title: HotROD location service version: "1.0" servers: - url: @{target} paths: /locations: get: summary: List locations, optionally filtered by name parameters: - name: name in: query schema: type: string example: Coffee responses: "200": description: Matching locations EOF # -I: low-risk warnings (like missing headers) don't fail the scan.  zap-api-scan.py -t /zap/wrk/openapi.yaml -f openapi -I \ -r zap-report.html -J zap-report.json # Fail the Job if ZAP found anything High risk.  HIGH=$(jq '[.site[].alerts[] | select(.riskcode == "3")] | length' zap-report.json) if [ "$HIGH" -gt 0 ]; then echo echo "ZAP found $HIGH High-risk issue(s):" jq -r '.site[].alerts[] | select(.riskcode == "3") | " \(.name) [\(.pluginid)]: param \(.instances[0].param), attack \(.instances[0].attack)"' zap-report.json exit 1 fi echo "No High-risk issues found." uploadArtifact: - path: /zap/wrk/zap-report.html - path: /zap/wrk/zap-report.json 

It does three things:

  • It scans only what changed. The OpenAPI definition lists only GET /locations and its name parameter. ZAP's API scan sends attack payloads to each operation in the definition, so the scan covers exactly what changed and leaves out operations that write data.
  • It reaches the change through the routing key. routingContext ties the Job to the sandbox, and with injectRoutingKey: auto a proxy next to the Job adds the routing headers to its outbound plain HTTP traffic. ZAP needs no extra configuration, and its own header settings stay free for your app's credentials if the scan needs to log in.
  • It fails on High-risk findings only. Low-risk findings, like missing security headers, show up in the report but don't fail the Job.

Run the scan:

signadot job submit -f .signadot/security/zap-api-scan.yaml \ --set sandbox=location-search \ --set target=http://location.hotrod:8081 \ --attach 

target is the baseline service's address. ZAP never needs to know where your change runs. After about 30 seconds, the Job fails:

WARN-NEW: SQL Injection [40018] x 1 http://location.hotrod:8081/locations?name=%27 (500 Internal Server Error) ... FAIL-NEW: 0 FAIL-INPROG: 0 WARN-NEW: 4 WARN-INPROG: 0 INFO: 0 IGNORE: 0 PASS: 116 ZAP found 1 High-risk issue(s): SQL Injection [40018]: param name, attack ' Error: exit status 1 

ZAP sent a single quote as name, and the service answered with a server error. A harmless value didn't cause one. Send the same request yourself to see why:

curl -H "baggage: sd-routing-key=$ROUTING_KEY" "http://location.hotrod:8081/locations?name='" 
Error 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ''' at line 1 

The quote ended the SQL string early. Whatever follows it in name runs as SQL, and the error message gives away the database. The full report has a description and a fix for each finding. Download it with the Job name from the output above:

signadot artifact download --job <job-name> /zap/wrk/zap-report.html 

You can also open both reports from the Job's page in the Signadot dashboard.

6. Fix it and scan again

The fix passes name to the database as a query parameter instead of pasting it into the SQL. It's one commit on the zap-location-search-fix branch:

query := "SELECT id, name, coordinates FROM locations" var args []any if name != "" { query += " WHERE name LIKE ?" args = append(args, "%"+name+"%") } rows, err := d.db.Query(query, args...) 

Switch to that branch:

git checkout zap-location-search-fix 

In the second terminal, stop the service with Ctrl+C and start it again:

go run ./cmd/hotrod location 

Scan again:

signadot job submit -f .signadot/security/zap-api-scan.yaml \ --set sandbox=location-search \ --set target=http://location.hotrod:8081 \ --attach 

This time the Job passes:

FAIL-NEW: 0 FAIL-INPROG: 0 WARN-NEW: 2 WARN-INPROG: 0 INFO: 0 IGNORE: 0 PASS: 117 No High-risk issues found. 

The single quote is now just a character to search for. It finds Rachel's Floral Designs:

curl -H "baggage: sd-routing-key=$ROUTING_KEY" "http://location.hotrod:8081/locations?name='" 

The change is ready for review, and the passing scan is evidence that it's safe to merge.

Scan a change deployed in the cluster

The sandbox decides where the change runs, so the Job works the same way when the change is deployed in the cluster. Build an image from your branch, then create a sandbox that forks the location Deployment with that image in place of local:

name: location-search spec: cluster: "@{cluster}" forks: - forkOf: kind: Deployment namespace: hotrod name: location customizations: images: - image: "@{image}" 

Then run the same signadot job submit command.

Run the scan as a pull request check

In CI, the pipeline builds an image from the pull request and creates a sandbox that forks the service's Deployment with that image, like the one above. Then it submits the same Job with --attach. If the scan finds a High-risk issue, the Job fails and so does the PR check, and both reports stay attached to the Job for the reviewer. See CI/CD integrations for running Signadot Jobs from GitHub Actions, GitLab, Bitbucket and Jenkins.

A coding agent can run the same loop on its own changes before it opens a pull request. See Run it in the agent loop and in CI.

Keep active scans contained

An API scan sends real attack payloads. The sandboxed service processes them, and its calls to shared services and databases are real. This tutorial keeps that safe by listing only a read-only operation in the OpenAPI definition. For your own services:

  • List only the operations the change touches in the scan's OpenAPI definition.
  • Before scanning operations that write data, give the sandbox its own database with a resource plugin, so injected rows never land in shared data.
  • To check a web UI without sending attacks, use ZAP's passive baseline scan, zap-baseline.py, in the same Job.

See Keep active scans contained for more.

Clean up

Stop the location service in the second terminal with Ctrl+C. Then run:

signadot sandbox delete location-search signadot jobrunnergroup delete zap signadot local disconnect 

To remove HotROD, run kubectl delete namespace hotrod.

Troubleshooting

What you see What to check
namespace "signadot-tests" does not exist when creating the runner group Run kubectl create namespace signadot-tests first.
The Job stays queued The runner pod is still pulling ZAP's image. Check kubectl -n signadot-tests get pods.
curl: Could not resolve host: location.hotrod Run signadot local status, and signadot local connect if it isn't connected.
The scan passes before the fix, or the routed curl returns all five locations Requests aren't reaching your machine. Check that go run ./cmd/hotrod location is running and that signadot sandbox get location-search shows the sandbox as ready.

Frequently asked questions

How do I run a ZAP scan against a single microservice change in Kubernetes?

Run the changed service in a Signadot sandbox and run ZAP as a Signadot Job in the cluster. The Job's routingContext ties it to the sandbox, and trafficManager.injectRoutingKey: auto adds the sandbox's routing key to every plain HTTP request ZAP sends. ZAP targets the baseline service's address, and Signadot routes its requests to the changed workload, so every finding belongs to that change.

Which ZAP scan should I run against a sandbox?

Use the API scan (zap-api-scan.py) for APIs, with an OpenAPI definition that lists only the operations the change touches. Use the baseline scan (zap-baseline.py), which runs passive rules only, for web UIs or any sandbox whose data isn't isolated. Use the full scan (zap-full-scan.py) for web apps when the sandbox has its own data.

Is it safe to run an active ZAP scan against a sandbox?

An active scan sends real attack payloads, and the sandboxed service's calls to shared services and databases are real. Keep it contained: list only the changed operations in the scan's OpenAPI definition, leave out operations that write data, and give the sandbox its own database with a resource plugin before scanning operations that do.

Can the ZAP scan run as a pull request check?

Yes. In CI, create a sandbox that forks the service Deployment with the image built from the pull request, then submit the same Job with signadot job submit --attach. The Job fails when ZAP reports a High-risk finding, which fails the PR check, and the HTML and JSON reports stay attached to the Job for reviewers.

Can ZAP scan code running on my laptop?

Yes. With a local sandbox and signadot local connect, requests that carry the sandbox routing key reach the service running on your machine. ZAP still runs in the cluster and targets the baseline address, so you fix the code, restart the service and scan again without building or pushing an image.

See also