Fastly Timeouts Caused By Request Collapsing

Published: January 7, 2021

Recently I was involved in diagnosing an odd issue. For a specific page on a website, customers were experiencing extremely slow responses, frequently timing out with the following message.

Timed out while waiting on cache-dca17735-DCA

NOTE: cache-dca17735-DCA in this case refers to Fastly's DCA data center in Ashburn, VA. (See: https://www.fastly.com/release-notes/q2-17-q3-17). This aspect of the response would change from user to user, however the "Timed out while waiting on" part was consistent.

However, when we looked at the application backend we saw no sign that it was unhealthy in any way. While we did see that throughput was slightly elevated for the route in question, average server response times were quick, and there were no signs that any of the infrastructure was overloaded.

We reported the issue to Fastly support, who advised that it was likely caused Request collapsing. I’m not 100% sure of the conditions that can lead to this happening but, the following were at play for us:

Apparently request collapsing creates a queue for requests, which can become backed up if a specific URL gets enough traffic under the right conditions.

The fix they suggested was to add a return (pass) targeting this specific URL to the vcl_recv subroutine.

sub vcl_recv {
    if (req.url.path ~ "/problem-url") {
        return(pass);
    }
}

NOTE: For Magento users this can be done via a Custom VCL snippet

Upon implementing this change we found that the issue no longer occured.


Max Chadwick Hi, I'm Max!

I'm a software developer who mainly works in PHP, but loves dabbling in other languages like Go and Ruby. Technical topics that interest me are monitoring, security and performance. I'm also a stickler for good documentation and clear technical writing.

During the day I lead a team of developers and solve challenging technical problems at Something Digital where I mainly work with the Magento platform. I've also spoken at a number of events.

In my spare time I blog about tech, work on open source and participate in bug bounty programs.

If you'd like to get in contact, you can find me on Twitter and LinkedIn.