Benchmarking the Impact of Implementing a CDN

Published: August 3, 2017

Tags:

In a talk I’m preparing titled, “Imagining A World Without Caching”, I’m benchmarking the impact of many different forms of caching. One type of caching that the talk covers is “edge caching” a.k.a. content delivery networks (CDNs). I spent a lot of time on Google trying to find hard data showing the impact that implementing a CDN had on page load times. Unfortunately, after nearly an hour of Googling, I couldn’t find the data I was looking for.

As such, I decided to do my own benchmarking. Here, I’ll share my data.

The CDN

The purpose of this investigation was not to compare one CDN to another, but rather to understand the impact of implementing any CDN. That being said, it is important to note that Cloudflare was used for this benchmarking.

The Origin

It is also relevant to know the origin in use during this benchmarking. In this case, the origin was GitHub Pages. A records for GitHub Pages point to 192.30.252.153 and 192.30.252.154. According to MaxMind those addresses are located in San Francisco.

The Test Location

The impact of a CDN is much greater when the distance of the user making the request is very far from the origin. As such, all benchmarking was done from Australia. WebPagetest was used to generate the requests.

Benchmarking Properly

It is important to remove variables that differ in how the origin or the CDN serve responses. For example, in this blog post the author draws the conclusion that a CDN is not beneficial. However, the underlying problem is that the CDN the author used was incapable of using SPDY, which he could use from his origin server.

In our case, both the origin, and the CDN are capable of using HTTP/2.

Additionally, each test variation was run 5 times and the totals were averaged to mitigate against anomalies.

Requesting A Page on This Blog

I started out by doing some tests linking to a post on this blog. The requests were made over a Cable (5/1 Mbps 28ms RTT) connection.

Here’s the data…

Cache Status	First Byte	Start Render	Speed Index	Load Time	Links
Miss	0.840	1.725	2779	3.020	1 / 2 / 3 / 4 / 5
Hit	0.800 (-4.79%)	1.514 (-12.25%)	2465 (-11.30%)	2.507 (-15.12%)	1 / 2 / 3 / 4 / 5

We can see that while time-to-first-byte is more or less unchanged, we experienced a modest reduction in the start render time, the speed index and the load time thanks to the CDN.

Requesting A Page From An Ecommerce Website

The post on this blog used for benchmarking loads significantly less assets than something like an ecommerce website would. As such, I next decided to test a sample ecommerce page.

I used wget to capture the static HTML and page requisites for the “Men’s Jackets” product listing page in an out of box Magento 2 installation with sample data. So how did that play out?

Cache Status	First Byte	Start Render	Speed Index	Load Time	Fully Loaded[1]	Link
Miss	0.890	2.733	2857	4.398	4.824	1 / 2 / 3 / 4 / 5
Hit	0.8432 (-5.24%)	2.325 (-14.92%)	2475 (-13.37%)	2.748 (-37.51%)	2.854 (-40.83%)	1 / 2 / 3 / 4 / 5

Here, we can see that while, the reduction in metrics like time-to-first-byte, start render and speed index are about the same as the blog post, load time and fully loaded show a much more significant decrease as a result of the CDN.

1. Note that fully loaded was intentionally omitted from the results table for my blog post. This is because of scripts injected into the page by Disqus and Google Analytics which I have no control over. However, in the case of the ecommerce page test, there are no 3rd party scripts at play.

What If We Cache The HTML Document?

In both the test cases above, Cloudflare not configured to cache HTML documents - which is it’s default configuration. So next, I decided to explore the impact of the CDN if the HTML document is additionally served from cache. Here are the results for the blog post again…

Cache Status	First Byte	Start Render	Speed Index	Load Time	Link
Miss	0.840	1.725	2779	3.020	1 / 2 / 3 / 4 / 5
Hit	0.433 (-48.48%)	1.213 (-29.69%)	2108 (-24.13%)	2.060 (-31.79%)	1 / 2 / 3 / 4 / 5

As you can see, serving the HTML from cache greatly improves the impact of a CDN. This is mainly because it reduces the time-to-first-byte (because the request does not need to reach the origin for the HTML document).

What About Choppy Network Conditions?

In all the above benchmarks, the request was made with WebPagetest’s Cable (5/1 Mbps 28ms RTT) connection. The final question I had was how much of a difference does a CDN make with choppy network conditions. I decided to re-run the test with WebPagetest’s Mobile 3G (1.6 Mbps/768 Kbps 300ms RTT) connection. I used the blog post again, and kept HTML document caching on. Here are the results…

Cache Status	First Byte	Start Render	Speed Index	Load Time	Link
Miss	1.960	3.226	5332	5.838	1 / 2 / 3 / 4 / 5
Hit	1.500 (-23.47%)	2.266 (-30.49%)	4076 (-24.53%)	4.672 (-20.26%)	1 / 2 / 3 / 4 / 5

The savings of a CDN over choppy network conditions were not exponentially greater (as I thought they might be), but rather proportional to the savings over cable for start render, speed index and load time. However, time-to-first-byte was reduced by roughly the same amount (~400ms) leading to a less significant impact, percentage-wise.

Conclusions

Based on my testing I have drawn the following conclusions…

The more assets your webpage requires (images, styles, scripts) the greater the impact you’ll see using a CDN
If you can serve the entire response from the edge, including the HTML document, you’ll see much more dramatic impact in terms of time-to-first-byte, start render AND speed index
The amount of improvement is more-or-less the same regardless of network conditions - slow network conditions will not see exponential improvements by using a CDN

Hope you found this data interesting / useful. If you think I got something wrong or have anything else to say about this, let me know in the comments below.