60 green checks. One old copy your test never reached.

Read the short version

Sixty successful checks did not reach every server. In our constructed loopback experiment, a client reused one connection and received sixty successful replies from the newer software. Opening fresh connections reached an older copy too.

That is why a test should record which running copies it reached, rather than just count successful requests. The experiment models connection reuse; it is not a claim about a customer incident or a production network.

If you own a service, compare the test results with your live inventory. Check each version directly and test how real requests are distributed. The full method, response records and limitations remain below.

Jump to a section

Sixty requests came back healthy. Every one reported the fixed version. One old copy was still there.

The check hadn’t lied. We had asked it a question it couldn’t answer.

Our test kept one network connection open. The router sent that connection to one fixed copy, so all sixty requests met the same copy. The other five weren’t examined at all. One of them reported the old version.

I’d call that a failed coverage test, even though the request dashboard looked perfect. The important number wasn’t sixty successes. It was one of six copies inspected.

Open the full-size chart (new tab)

A plain-English version

Imagine six branches behind one customer-service number. You call sixty times, but the system keeps reconnecting you to the first branch. Every answer is correct. That proves something about that branch. It tells you very little about the other five.

Network connections can create a similar blind spot. Reusing a connection is normal and often useful. The mistake is treating repeated answers as proof that every copy was reached.

This matters for an AI application too. The part of your system that calls a model, retrieves documents or executes an approved action may run in several copies. A successful request through the public address doesn’t, on its own, establish the software version of every copy behind it.

What we actually built

Six small HTTP servers ran on this computer’s loopback interface. Copies 1–5 returned version: fixed. Copy 6 returned version: old. All six returned HTTP 200 and health: healthy. These are deliberately chosen labels, not actual vulnerable or patched packages.

A TCP proxy picked the next server whenever it accepted a new connection, then relayed that connection to the same server until it closed. Selection was deterministic round robin. The first connection deliberately went to fixed copy 1. We used Python’s standard library and real sockets; the recorded response rows came from actual HTTP exchanges.

We compared three methods:

1. One reused connection: sixty requests without closing the client’s connection.
2. Fresh connections: sixty requests, each opening and closing its own connection.
3. Direct inventory: one request to each of the six known server addresses, bypassing the proxy.

The ten old responses were a consequence of our routing rule: sixty fresh connections made ten complete rounds across six servers. They are not an estimate of the chance that a real user will hit an old server.

What happened after the change

We changed copy 6’s in-memory version response from old to fixed and repeated all three checks. All 126 responses in this second phase reported fixed. The direct inventory named all six copies. The fresh-connection batch reached all six too.

The reused connection still reached just one copy, this time copy 2. Its all-clear result happened to agree with the complete inventory. Its coverage was still inadequate to prove that conclusion on its own.

That distinction is worth keeping. An unreliable verification method doesn’t become reliable because the latest result happens to be right.

Return to contents

Reproduce it without a cluster

You need Python 3 and a local folder. There are no paid APIs, cloud resources or third-party packages. Download the script, inspect it, then run it in that folder:

Download connection_lab.py

python connection_lab.py

The script creates a run folder beside itself, writes the responses and summary, and prints the complete JSON result. Running it again overwrites those lab output files. All listeners bind to 127.0.0.1 on temporary ports. It closes its sockets and threads before reporting success.

On our retained run, these fields were present in the real output (excerpt; not the entire JSON):

"response_rows": 252,
"proxy_connections": 122,
"assertions_passed": true,
"owned_threads_and_sockets_closed": true

Read the full captured output · Inspect the structured result · Download the response CSV

We also reran the experiment and compared the CSV rows with the aggregate counts. The assertions check HTTP success, expected identities, distinct-copy coverage, routing counts and cleanup. The result records a UTC run timestamp so you can distinguish our captured run from your own.

Five questions for your team

You don’t need to know Python to ask for stronger evidence. Send this list to whoever owns the rollout:

1. How many copies should exist? Ask for the expected inventory, including workers, standby environments and separately managed instances where relevant.
2. Which copies did this test actually reach? Ask for unique instance identities. A request count is not an instance count.
3. What version did each running copy report? Match identities to a known approved build or package version. A friendly “healthy” response isn’t that evidence.
4. What could make the inventory incomplete? Check alternate traffic paths, old groups, failover targets and jobs outside the main application.
5. What evidence closes the change? A dated list of expected copies, observed versions, exceptions and a named owner. Resolve missing or old copies before calling the rollout verified.

For the actual Kubernetes rollout lab, traffic test and read-only Service checker, see the original stale-copy investigation. This smaller experiment answers a different question: how connection reuse can weaken the coverage of a traffic sample.

Where this result stops

This was engineered to expose one mechanism. Starting the reused connection on copy 6 would have found the old version immediately, while still checking only one copy. Changing the routing policy, adding retries or using a different protocol can change the outcome.

Fresh connections do not guarantee full coverage in production. Session affinity, uneven weights, topology and additional routing layers can send them to the same subset. Direct inventory is stronger in our lab because the inventory is complete and each address names a separate server. Real inventories can be wrong, and reported version labels need to correspond to the executable build you care about.

We did not install a security update, test an exploit, measure latency or prove anything about a specific production load balancer. Python’s HTTP client documentation explains the client interface used here. Kubernetes’ Service documentation describes its own service and endpoint abstractions; our loopback proxy is not a substitute for testing that environment.

Count the copies you verified, not just the requests that passed.

More security investigations · Plan a controlled AI pilot · Get useful research updates