The rollout passed.
One copy still ran the old software.

We staged a rollout with six copies of an app. Kubernetes reported that the update succeeded. All six copies answered the health check. One was still running the old software.

The older copy belonged to a separate group that the update never touched. It remained on the list of copies that could take requests. That’s the stale-copy problem. You can get a clean result for the group you updated while customers can still reach an older copy.

We used a real Brotli security update for the lab. Brotli unpacks compressed data. OpenAI says GPT‑6 Cyber helped its researchers demonstrate what the flaw could do. The public records show which Node.js and .NET releases carry the fix. The lab shows what to check after you install one. See the release records.

$ kubectl --context default -n brotli-rollout-lab set image deployment/api api=node:24.22.1-alpine
deployment.apps/api image updated
$ kubectl --context default -n brotli-rollout-lab rollout status deployment/api --timeout=180s
deployment "api" successfully rolled out

READY_ENDPOINT HTTP NODE
api-5cb5f6cd9f-8kh4v 200 v24.22.1
api-5cb5f6cd9f-97l7b 200 v24.22.1
api-5cb5f6cd9f-bls5j 200 v24.22.1
api-5cb5f6cd9f-xrmkf 200 v24.22.1
api-5cb5f6cd9f-zzsj4 200 v24.22.1
api-legacy-56f9f9cbf4-zhntw 200 v24.22.0

$ kubectl --context default -n brotli-rollout-lab set image deployment/api-legacy api=node:24.22.1-alpine
deployment.apps/api-legacy image updated
$ kubectl --context default -n brotli-rollout-lab rollout status deployment/api-legacy --timeout=180s
deployment "api-legacy" successfully rolled out

READY_ENDPOINT HTTP NODE
api-5cb5f6cd9f-8kh4v 200 v24.22.1
api-5cb5f6cd9f-97l7b 200 v24.22.1
api-5cb5f6cd9f-bls5j 200 v24.22.1
api-5cb5f6cd9f-xrmkf 200 v24.22.1
api-5cb5f6cd9f-zzsj4 200 v24.22.1
api-legacy-85456dcd77-h8c22 200 v24.22.1

$ kubectl --context default -n brotli-rollout-lab get endpointslice -l kubernetes.io/service-name=api -o go-template=\{\{range\ .items\}\}\{\{range\ .endpoints\}\}\{\{if\ .conditions.ready\}\}\{\{.targetRef.name\}\}\{\{\"\\t\"\}\}\{\{index\ .addresses\ 0\}\}\{\{\"\\n\"\}\}\{\{end\}\}\{\{end\}\}\{\{end\}\}
$ kubectl --context default -n brotli-rollout-lab exec api-5cb5f6cd9f-8kh4v -- node -e const\ \[ip\,\ expectedPod\]\ =\ process.argv.slice\(1\)\;\ fetch\(\"http://\"\ +\ ip\ +\ \":3000/health\"\,\ \{signal:\ AbortSignal.timeout\(5000\)\}\).then\(async\ r\ =\>\ \{\ const\ body\ =\ await\ r.json\(\)\;\ if\ \(\!r.ok\ \|\|\ body.pod\ \!==\ expectedPod\)\ throw\ Error\(\"unexpected\ endpoint\ response\"\)\;\ console.log\(body.pod\ +\ \"\ \"\ +\ r.status\ +\ \"\ \"\ +\ body.node\)\;\ \}\).catch\(e\ =\>\ \{\ console.error\(e\)\;\ process.exitCode\ =\ 1\;\ \}\)\; 10.42.0.22 api-legacy-56f9f9cbf4-zhntw

$ kubectl --context default -n brotli-rollout-lab get pods -l app=api -o custom-columns=POD:.metadata.name\,TRACK:.metadata.labels.track\,IMAGE:.spec.containers\[0\].image\,IMAGE_ID:.status.containerStatuses\[0\].imageID --sort-by=.metadata.name
POD                           TRACK    IMAGE                 IMAGE_ID
api-legacy-56f9f9cbf4-zhntw   legacy   node:24.22.0-alpine   docker.io/library/node@sha256:44fce696f38ba8c987c4db89954051000bbd05ffd8d8659ba15ed4a8602e937f
api-legacy-85456dcd77-h8c22   legacy   node:24.22.1-alpine   docker.io/library/node@sha256:06d7adf3e0ef61a13f1d28682726366f97a15aa1888c934d3ecec86c33918b53

Five questions before you sign off

Say your booking or checkout service needs this update. Your provider says, “The rollout is green.” Before you close the ticket, find out whether that covers every copy of the service customers can reach. You can ask for the evidence without knowing how to read a Kubernetes command.

How to read the reply

“Five of six copies verified, one taken out of service pending an update, platform team rechecking at 4 p.m.” tells you how much work remains and who is doing it. Put that in the handoff as an open item. If your provider says the flaw does not apply, ask them to name the product or version they checked and what rules it out. On a hosted service, they may give you an advisory or a dated status statement instead of access to their infrastructure.

For your own laptop, the short version is to use your vendor’s update channel, restart when prompted, then check that the new version is running. See the personal-device steps. If you’re coordinating a team, turn the reply into a short action plan with a named owner and review date.

python3 service-image-check.py \
  --context YOUR_CONTEXT --namespace YOUR_NAMESPACE \
  --service YOUR_SERVICE --container YOUR_APP_CONTAINER \
  --digest YOUR_APPROVED_SHA256_DIGEST --min-pods 6

MISMATCH api-legacy-56f9f9cbf4-zkb87 sha256:44fce696f38ba8c987c4db89954051000bbd05ffd8d8659ba15ed4a8602e937f
FAIL: 1 of 6 ready Pods reported an unexpected image digest
Exit code: 1

PASS: 6 ready Pods reported approved image digests
Exit code: 0

INCONCLUSIVE: found 6 distinct eligible Pods; expected at least 7
Exit code: 2

$approved = '4.8.2'
$roster = @(Import-Csv -LiteralPath './expected-instances.csv' -ErrorAction Stop)
$reports = @(Import-Csv -LiteralPath './running-versions.csv' -ErrorAction Stop)
if (!$roster.Count -or (($roster + $reports) | Where-Object { !$_.Instance })) {
    throw 'The expected inventory is empty or an export contains an unnamed instance.'
}
$expected = @($roster.Instance | Sort-Object -Unique)
$byInstance = @{}
if ($reports.Count) {
    $byInstance = $reports | Group-Object -Property Instance -AsHashTable -AsString
}
foreach ($instance in $expected) {
    $rows = @($byInstance[$instance] | Where-Object { $_ })
    if (!$rows.Count) {
        [pscustomobject]@{Instance=$instance; Issue='No version report'; Version=''}
    } elseif ($rows.Count -ne 1 -or !$rows[0].Version) {
        [pscustomobject]@{Instance=$instance; Issue='Duplicate or blank report'; Version=''}
    } elseif ($rows[0].Version -cne $approved) {
        [pscustomobject]@{Instance=$instance; Issue='Version mismatch'; Version=$rows[0].Version}
    }
}
foreach ($instance in $byInstance.Keys | Sort-Object) {
    if ($instance -notin $expected) {
        [pscustomobject]@{Instance=$instance; Issue='Outside expected inventory'; Version=''}
    }
}

The conversation worth having

Start with one relevant advisory and one system the business relies on. The planner below turns what you know—and what you still need to ask—into a response brief, an action register and a focused meeting agenda.

NIST’s enterprise patch-management guidance includes identifying, prioritizing, acquiring, installing, and verifying updates. The meeting kit below is AI Vanguard’s practical way to organize that handoff; it is not a new security standard. [5]

This is especially important when there is evidence of active exploitation. The UK’s National Cyber Security Centre recommends a rapid response pathway and coordination with the people running affected systems. Urgency needs a route through the organization. [6]

Walk into the meeting with a plan.

Eight questions. A clear next move. A briefing your team can actually use.

For one advisory or security concern at a time. “I don’t know” is a useful answer—we’ll turn it into a question for the right person.

A brief worth bringing to the table.

Your situation. Specific actions. Owners and dates. A clear definition of done.

See an example PDFIllustrative answers, not a real company assessment.

Prefer to work on paper? Take the meeting brief, action register and evidence checklist with you.

Download the Patch Verification Meeting Kit (PDF)

The personalised planner needs JavaScript. The PDF works without it.

What this tool does—and where judgement matters

This is an editorial planning aid based on your answers. It does not scan systems, verify evidence, calculate breach probability or certify security. Your technical owner sets the response priority and deadlines. Suspected compromise should go through your organisation’s incident process.

The questions draw on the CISA vulnerability response playbook: establish applicability, prioritise, remediate and verify. This is not an official CISA assessment.

Your next action depends on what you run

If you just want your own devices looked after

Use the official update mechanism for your browser and operating system. Complete any requested restart. On a managed work device, check with IT before changing an update policy.

For a Mozilla-installed desktop Firefox, the About Firefox window checks for updates and offers a restart when needed. Microsoft Store and Linux package-manager installations use their respective update channels. Mozilla documents the differences in its update guide [7].

If you run a business but not its infrastructure

Send the action plan to the person who owns IT or to your service provider. Ask which relevant advisories affect your systems and which responsibilities belong to you. Request an answer with evidence, even when that answer is “not affected.”

Three quick answers