Our tracking API runs on eight EC2 instances behind an Application Load Balancer, deployed in place by CodeDeploy one instance at a time. Each instance is taken out of the target group, updated, health checked and put back. On a quiet day a release took about eight minutes and nobody had ever timed it more carefully than that.
In October a carrier changed an error format, and every status update from them was failing to parse. The fix was two lines, reviewed in ten minutes and merged at 14:20. It finished deploying at 15:16. For that hour depots were ringing customer service about vans the system said had not left.
Almost all of the time was spent waiting. When CodeDeploy blocks traffic to an instance, the load balancer stops sending it new requests and waits for the requests already in flight to finish before it completes deregistration. It waits at most the target group's deregistration delay, which defaults to three hundred seconds. A target with nothing in flight is released at once, which is why our early releases had been quick.
By October they were not. In the summer we had added a live board for the depot offices: a page that holds a server-sent events stream open to the API and receives each van's position as it changes. Every depot screen kept one request open, all day, and the load balancer spread them across all eight instances. Every instance therefore had a request that would never finish on its own, and every deregistration waited the full five minutes. One instance at a time, that was forty minutes of waiting on top of the work itself. The board had been slowing every release for three months, and it took a hurry for us to notice.
The deregistration delay is now thirty seconds, longer than any ordinary request we serve. A BeforeBlockTraffic hook tells the application to close its event streams, and the board's EventSource reconnects by itself within a few seconds, to an instance that is staying. Releases run two instances at a time, and the pipeline reports how long each step took, so the number we watch is time from merge to the last instance serving.
A deploy is as slow as the longest request it is willing to wait for. Ours was willing to wait for a request designed never to end.
– Sergey Shinder

