Four Systems, No Conversation
Draining is usually stated in one sentence: stop sending new requests, and finish the ones in flight. Both halves are races.
Removing an instance on purpose is not one action. It is several, started at roughly the same time by things that cannot see each other.
That is why the answer is always a delay and not a handshake: nobody has anywhere to send an acknowledgement.
The First Race: The Pool Update
One of them takes effect immediately and the other has to travel, which means the instance can be gone before the pool knows.
Nothing was misconfigured in any of the three runs. The only variable is how long the instance kept serving after being told to stop.
The removal takes 300 milliseconds to reach the balancer in all three runs, and the failures track the gap.
Both quantities are properties of your specific setup, and neither is in any documentation.
Measure the propagation delay instead of guessing it. Remove an instance, keep sending requests at a known rate, and count how many still arrive after the removal was issued. It is often larger than expected in a big cluster, because it depends on how many proxies have to be told.
This is not a bug in any orchestrator. There is no ordering guarantee between the two, and the documentation says so.
Which produces the standard advice, and it looks wrong until you see the ordering: on the shutdown signal, do not stop serving. Keep answering for longer than the endpoint change takes to propagate, and only then begin the real shutdown.