Magento Cloud Page Cache Hit Rate with New Relic Logs

Published: December 30, 2020


Way back in 2016, I spoke extensively about monitoring page cache hit rate in Magento. The content (and code) was based on the Magento 1 full page cache implementation. Now, four-and-a-half years later I wanted to give an update on the topic, discussing how we can now do this out of the box via New Relic logs for merchants running on Magento Cloud.

New Relic Logs

As discussed in my recent post Magento Cloud Response Times as measured by Fastly in New Relic Magento Cloud now sends Fastly logs to New Relic out of the box. These logs include the cache_status which will typically be either, HIT, MISS or PASS. While the it may be possible to retreive some data from the New Relic Log Search interface I find New Relic Insights, which supports NRQL queries to be much easier to use. Let’s take a look at some queries.

Global Hit Rate text/html on site

The below example query will help us visualize the cache hit rate for text/html URLs for the past 7 days.

WHERE cache_status IS NOT NULL
    AND origin_host LIKE 'www.%'
    AND content_type = 'text/html; charset=UTF-8'
SINCE 7 days ago
FACET cache_status
TIMESERIES 30 minutes

A few things to call out with this query

  • We ensure cache_status is not null to limit the Log data to only Fastly logs (other logs can also be queried FROM Log)
  • We are filtering origin_host to only include www.. This will filter out the staging environment for us, and also requests that are not to the www subdomain (e.g. redirected requests lacking www)
  • We add 30 minutes after TIMESERIES to add more precision to the graph. New Relic will otherwise aggregate data into 6 hours buckets when requesting 7 days of data.

Here’s a preview of the graph:

Graph of query result

If we want to get the actual percent we’ll need to remove TIMESERIES from the query. We can then click into the pie chart.

Screenshot showing the pie chart in the New Relic UI

By Route

While the global hit rate is interesting, on it’s own it not super actionable. For example, the checkout and cart page will never be cacheable. What’s more interesting it to know the hit rate by route (e.g. to see hit rate just for category pages, or product pages).

Unfortunately, Magento Cloud doesn’t currently support this. The implementation could be improved by Magento sending a response header containing the routing info, and updating the Fastly log format to additionally stream that to New Relic

Currently out best option is to drill in to specific URLs.

For example, we can see the hit rate for the home page by adding WHERE url = '/' to the query. We’ll also filter for only GET requests as there can be POST requests to / which will return PASS (on the client in question these seem to come from user agent MacOutlook for some reason).

Another graph of query result

Max Chadwick Hi, I'm Max!

I'm a software developer who mainly works in PHP, but loves dabbling in other languages like Go and Ruby. Technical topics that interest me are monitoring, security and performance. I'm also a stickler for good documentation and clear technical writing.

During the day I lead a team of developers and solve challenging technical problems at Something Digital where I mainly work with the Magento platform. I've also spoken at a number of events.

In my spare time I blog about tech, work on open source and participate in bug bounty programs.

If you'd like to get in contact, you can find me on Twitter and LinkedIn.