Jekyll2023-07-21T00:31:11+00:00https://maxchadwick.xyz/feed.xmlMax ChadwickMy WebsitemaxpchadwickAdd an IP Address to a Fastly ACL via the CLI with Magento2023-07-20T00:00:00+00:002023-07-20T00:00:00+00:00https://maxchadwick.xyz/blog/add-an-ip-to-fastly-acl-via-cli-magento<p>Recently I was in a bit of a pickle on a new Magento project that my company was taking over.</p>
<p>Access to the staging site was restricted via Fastly. I had SSH access to the environment, but my IP address was not allowed via the ACL, so I couldn’t connect to the website’s backend UI to grant myself access.</p>
<p>I wound up figuring out how to manage this via the CLI. Since I struggled a bit with figuring this out I figured I’d shared my findings here.</p>
<!-- excerpt_separator -->
<h3 id="the-endpoint-to-call">The Endpoint to Call</h3>
<p>IP addresses can be added to an ACL via the <a href="https://developer.fastly.com/reference/api/acls/acl-entry/#create-acl-entry">“Create an ACL entry”</a> resource.</p>
<p>The request looks like this</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>POST /service/[service_id]/acl/[acl_id]/entry
</code></pre></div></div>
<p>The IP address is then passed in the request body along with other parameters such as a comment</p>
<h3 id="figuring-out-the-service-id">Figuring Out The Service ID</h3>
<p>Assuming you are using Magento Cloud the Service ID (and Fastly Key) can be found in the <code class="language-plaintext highlighter-rouge">/mnt/shared/fastly_tokens.txt</code> file. “API Token” is the <code class="language-plaintext highlighter-rouge">FASTLY_KEY</code> and “Serivce ID” is the <code class="language-plaintext highlighter-rouge">SERVICE_ID</code>.</p>
<h3 id="finding-the-acl-id">Finding the ACL ID</h3>
<p>First, get the active version. You can do this as follows, assuming you have <code class="language-plaintext highlighter-rouge">jq</code> installed.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># Get the active version. In this example 105 is active
$ curl --silent -H "Fastly-Key: FASTLY_KEY" https://api.fastly.com/service/SERVICE_ID/version \
| jq '.[] | if .active then .number else empty end'
105
</code></pre></div></div>
<p>Next review the list of ACLs for that version</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ curl --silent -H "Fastly-Key: FASTLY_KEY" https://api.fastly.com/service/SERVICE_ID/version/VERSION/acl | jq
</code></pre></div></div>
<p>Here you will find the id of the ACL you want to append to</p>
<h3 id="adding-the-ip">Adding the IP</h3>
<p>You can certainly issue a curl request, but another option is to do this with <code class="language-plaintext highlighter-rouge">n98-magerun2 dev:console</code>, which is how I did it. The commands I ran looked like this…</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ XDG_CONFIG_HOME=~/var/ var/n98-magerun2.phar dev:console
>>> $api = $di->get('Fastly\Cdn\Model\Api')
>>> $api->upsertAclItem(ACL_ID, IP_TO_INSERT, null, COMMENT)
</code></pre></div></div>maxpchadwickRecently I was in a bit of a pickle on a new Magento project that my company was taking over. Access to the staging site was restricted via Fastly. I had SSH access to the environment, but my IP address was not allowed via the ACL, so I couldn’t connect to the website’s backend UI to grant myself access. I wound up figuring out how to manage this via the CLI. Since I struggled a bit with figuring this out I figured I’d shared my findings here.What CURLOPT_FAILONERROR does in PHP2023-04-12T00:00:00+00:002023-04-12T00:00:00+00:00https://maxchadwick.xyz/blog/curlopt-failonerror-php-behavior<div class="tout tout--secondary">
<p>Testing for this blog post was done with PHP version 8.2.1</p>
</div>
<p>During a recent code review I learned about <a href="https://curl.se/libcurl/c/CURLOPT_FAILONERROR.html"><code class="language-plaintext highlighter-rouge">CURLOPT_FAILONERROR</code></a> for the first time.</p>
<p>I read through both the <a href="https://curl.se/libcurl/c/CURLOPT_FAILONERROR.html">libcurl documentation</a> as well as <a href="https://www.php.net/manual/en/function.curl-setopt.php">the PHP documentation</a> and in the end was still unclear exactly what this option does.</p>
<p>In this post I’ll share my findings from some experimentation.</p>
<!-- excerpt_separator -->
<h3 id="what-the-libcurl-documentation-says">What the libcurl documentation says</h3>
<p>The libcurl documentation describes the behavior as follows:</p>
<blockquote>
<p>Request failure on HTTP response >= 400</p>
<p>A long parameter set to 1 tells the library to fail the request if the HTTP code returned is equal to or larger than 400. The default action would be to return the page normally, ignoring that code.</p>
<p>When this option is used and an error is detected, it will cause the connection to get closed and CURLE_HTTP_RETURNED_ERROR is returned.</p>
</blockquote>
<p>Source: <a href="https://curl.se/libcurl/c/CURLOPT_FAILONERROR.html">https://curl.se/libcurl/c/CURLOPT_FAILONERROR.html</a></p>
<h3 id="what-the-php-documentation-says">What the PHP documentation says</h3>
<blockquote>
<p><code class="language-plaintext highlighter-rouge">true</code> to fail verbosely if the HTTP code returned is greater than or equal to 400. The default behavior is to return the page normally, ignoring the code.</p>
</blockquote>
<p>Source: <a href="https://www.php.net/manual/en/function.curl-setopt.php">https://www.php.net/manual/en/function.curl-setopt.php</a></p>
<h3 id="my-questions">My Questions</h3>
<p>Reading through these pieces of documentation I was left with a few questions</p>
<ul>
<li>What is the definition “fail the request” (from the libcurl documentation) or “fail” from the PHP documentation?</li>
<li>What will <code class="language-plaintext highlighter-rouge">curl_exec</code> return if the request “fails”? The libcurl documentation seems to suggest the return value will be <code class="language-plaintext highlighter-rouge">CURLE_HTTP_RETURNED_ERROR</code>.</li>
</ul>
<h3 id="first-order-of-business-what-is-curle_http_returned_error">First order of business: What is <code class="language-plaintext highlighter-rouge">CURLE_HTTP_RETURNED_ERROR</code>?</h3>
<p>The first thing I was interested in was what the <code class="language-plaintext highlighter-rouge">CURLE_HTTP_RETURNED_ERROR</code> <code class="language-plaintext highlighter-rouge">const</code> evaluated to in PHP. Using <code class="language-plaintext highlighter-rouge">php -r</code> here’s what I found:</p>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">$</span> <span class="n">php</span> <span class="o">-</span><span class="n">r</span> <span class="s1">'var_dump(CURLE_HTTP_RETURNED_ERROR) . PHP_EOL;'</span>
<span class="nf">int</span><span class="p">(</span><span class="mi">22</span><span class="p">)</span>
<span class="err">$</span> <span class="n">php</span> <span class="o">-</span><span class="n">r</span> <span class="s1">'var_dump((bool)CURLE_HTTP_RETURNED_ERROR) . PHP_EOL;'</span>
<span class="nf">bool</span><span class="p">(</span><span class="kc">true</span><span class="p">)</span>
</code></pre></div></div>
<p>Given these findings it seemed unlikely to me that PHP would actually return <code class="language-plaintext highlighter-rouge">CURLE_HTTP_RETURNED_ERROR</code> if the request “failed” (would it <em>really</em> return a truth-y value?), despite what the <code class="language-plaintext highlighter-rouge">libcurl</code> documentation had to say.</p>
<h3 id="running-some-tests">Running some tests</h3>
<p>First, I created a simple script that would return an HTTP 503 response code and a response body with the string “error”.</p>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><?php</span>
<span class="nb">http_response_code</span><span class="p">(</span><span class="mi">503</span><span class="p">);</span>
<span class="k">echo</span> <span class="s1">'Error'</span> <span class="mf">.</span> <span class="kc">PHP_EOL</span><span class="p">;</span>
</code></pre></div></div>
<p>Next I used <code class="language-plaintext highlighter-rouge">php -S</code> to run a server on port 1234 that would execute that script.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ php -S localhost:1234
[Wed Apr 12 21:30:36 2023] PHP 8.2.1 Development Server (http://localhost:1234) started
</code></pre></div></div>
<p>Then, I created a script that would send a request to my server, to test the behavior of <code class="language-plaintext highlighter-rouge">CURLOPT_FAILONERROR</code>.</p>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><?php</span>
<span class="nv">$ch</span> <span class="o">=</span> <span class="nb">curl_init</span><span class="p">();</span>
<span class="nb">curl_setopt</span><span class="p">(</span><span class="nv">$ch</span><span class="p">,</span> <span class="no">CURLOPT_URL</span><span class="p">,</span> <span class="s1">'http://localhost:1234'</span><span class="p">);</span>
<span class="nb">curl_setopt</span><span class="p">(</span><span class="nv">$ch</span><span class="p">,</span> <span class="no">CURLOPT_RETURNTRANSFER</span><span class="p">,</span> <span class="mi">1</span><span class="p">);</span>
<span class="nb">curl_setopt</span><span class="p">(</span><span class="nv">$ch</span><span class="p">,</span> <span class="no">CURLOPT_FAILONERROR</span><span class="p">,</span> <span class="kc">true</span><span class="p">);</span>
<span class="nv">$response</span> <span class="o">=</span> <span class="nb">curl_exec</span><span class="p">(</span><span class="nv">$ch</span><span class="p">);</span>
<span class="nb">var_dump</span><span class="p">(</span><span class="nv">$response</span><span class="p">)</span> <span class="mf">.</span> <span class="kc">PHP_EOL</span><span class="p">;</span>
<span class="nb">var_dump</span><span class="p">(</span><span class="nb">curl_error</span><span class="p">(</span><span class="nv">$ch</span><span class="p">)</span> <span class="mf">.</span> <span class="kc">PHP_EOL</span><span class="p">);</span>
</code></pre></div></div>
<p>Finally, I got the response.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ php test.php
bool(false)
string(38) "The requested URL returned error: 503
"
</code></pre></div></div>
<h3 id="what-i-learned">What I learned</h3>
<p>Based on this test, here’s what I learned about what <code class="language-plaintext highlighter-rouge">CURLOPT_FAILONERROR</code> does in PHP</p>
<ul>
<li>If the URL returns an HTTP status code >= 400, <code class="language-plaintext highlighter-rouge">curl_exec</code> will return <code class="language-plaintext highlighter-rouge">false</code>
<ul>
<li><code class="language-plaintext highlighter-rouge">CURLOPT_RETURNTRANSFER</code> cannot be used to access the response body when this option is used</li>
</ul>
</li>
<li><code class="language-plaintext highlighter-rouge">curl_error</code> will return a message indicating the HTTP status code that was received.</li>
</ul>
<h3 id="my-conclusion">My conclusion</h3>
<p>Ultimately <code class="language-plaintext highlighter-rouge">CURLOPT_FAILONERROR</code> doesn’t seem like a great option since it discards the response body, which might have useful information in the case of a failure. The <code class="language-plaintext highlighter-rouge">CURLINFO_HTTP_CODE</code> option of <code class="language-plaintext highlighter-rouge">curl_getinfo</code> seems like a better way the handle error unexpected HTTP response codes.</p>maxpchadwickTesting for this blog post was done with PHP version 8.2.1 During a recent code review I learned about CURLOPT_FAILONERROR for the first time. I read through both the libcurl documentation as well as the PHP documentation and in the end was still unclear exactly what this option does. In this post I’ll share my findings from some experimentation.Experimenting with Partytown2022-12-22T00:00:00+00:002022-12-22T00:00:00+00:00https://maxchadwick.xyz/blog/experimenting-with-partytown<p>A couple weeks back in a Twitter conversation I learned about <a href="https://partytown.builder.io/">Partytown</a>.</p>
<blockquote class="twitter-tweet"><p lang="en" dir="ltr">Would you try to leverage the script off the main thread with Partytown?<br />Didn't use myself yet. Curious about your result.<a href="https://t.co/49ifl6QX7I">https://t.co/49ifl6QX7I</a></p>— Denis Metzler (@DenisMetzler) <a href="https://twitter.com/DenisMetzler/status/1602584859104481282?ref_src=twsrc%5Etfw">December 13, 2022</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>I was immediately intrigued by the idea and saw great potential, especially on the ecommerce projects that I work on on a daily basis.</p>
<p>Today I spent some time playing with Partytown. In this post I’ll share my process and findings.</p>
<!-- excerpt_separator -->
<h3 id="methodology">Methodology</h3>
<p>For my use-case I was looking to quickly test some potential performance optimizations against a remote website that I didn’t have a copy of running locally. The website was running Adobe Commerce (a.k.a. Magento) on the Adobe Commerce Cloud infrastructure.</p>
<p>Running this type of testing is a task that comes up somewhat frequently for me and my current favorite way to do this is using Chrome developer tools <a href="https://developer.chrome.com/blog/new-in-devtools-65/#overrides">local overrides</a>.</p>
<p>Partytown publishes some <a href="https://partytown.builder.io/html">documentation</a> on how to integrate into plain old HTML although I did find it a bit confusing. Here’s what I wound up doing.</p>
<h3 id="get-a-copy-of-the-partytown-code">Get a copy of the Partytown code</h3>
<p>First we get a copy of the Partytown code</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># Make a random temporary folder
mkdir test && cd test
# Install Partytown
npm install @builder.io/partytown
# Use partytown copylib to obtain the code for web publishing
# Partytown wants to use ~partytown as the directory name
# but the ~ in the folder name needs to be escaped and is annoying
node node_modules/@builder.io/partytown/bin/partytown.cjs copylib partytown
</code></pre></div></div>
<p>Now your <code class="language-plaintext highlighter-rouge">partytown</code> directory should look something like this…</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>partytown
├── debug
│ ├── partytown-atomics.js
│ ├── partytown-media.js
│ ├── partytown-sandbox-sw.js
│ ├── partytown-sw.js
│ ├── partytown-ww-atomics.js
│ ├── partytown-ww-sw.js
│ └── partytown.js
├── partytown-atomics.js
├── partytown-media.js
├── partytown-sw.js
└── partytown.js
</code></pre></div></div>
<h3 id="upload-the-code-to-the-remote-server">Upload the code to the remote server</h3>
<p>Per the Partytown documentation, the code needs to be hosted from the origin domain. I attempted to simpy drop my <code class="language-plaintext highlighter-rouge">partytown</code> directory into my local overrides folder for the website, however Chrome was complaining that the service worker file wasn’t being loaded with the correct MIME type. To get around this I just uploaded the files to the remote server (maybe there’s a better way to do this?).</p>
<p>On Adobe Commerce Cloud was can drop them into the <code class="language-plaintext highlighter-rouge">pub/media</code> folder.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># Zip up the code
zip -r partytown.zip partytown
# scp it up
scp partytown.zip user@host:~/pub/media
# SSH in and unzip it
ssh user@host
cd pub/media
unzip partytown
</code></pre></div></div>
<h3 id="add-the-partytown-snippet-to-the-head">Add The Partytown Snippet to the head</h3>
<p>Using local overrides inline the snippet into the <code class="language-plaintext highlighter-rouge"><head></code> of the document. This will look something like this.</p>
<p>Note that we have to manually replace <code class="language-plaintext highlighter-rouge">/~partytown</code> with <code class="language-plaintext highlighter-rouge">/media/partytown</code>.</p>
<div class="language-html wrap highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><script></span>
<span class="cm">/* Partytown 0.7.3 - MIT builder.io */</span>
<span class="o">!</span><span class="kd">function</span><span class="p">(</span><span class="nx">t</span><span class="p">,</span><span class="nx">e</span><span class="p">,</span><span class="nx">n</span><span class="p">,</span><span class="nx">i</span><span class="p">,</span><span class="nx">r</span><span class="p">,</span><span class="nx">o</span><span class="p">,</span><span class="nx">a</span><span class="p">,</span><span class="nx">d</span><span class="p">,</span><span class="nx">s</span><span class="p">,</span><span class="nx">c</span><span class="p">,</span><span class="nx">p</span><span class="p">,</span><span class="nx">l</span><span class="p">){</span><span class="kd">function</span> <span class="nx">u</span><span class="p">(){</span><span class="nx">l</span><span class="o">||</span><span class="p">(</span><span class="nx">l</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span><span class="dl">"</span><span class="s2">/</span><span class="dl">"</span><span class="o">==</span><span class="p">(</span><span class="nx">a</span><span class="o">=</span><span class="p">(</span><span class="nx">o</span><span class="p">.</span><span class="nx">lib</span><span class="o">||</span><span class="dl">"</span><span class="s2">/media/partytown/</span><span class="dl">"</span><span class="p">)</span><span class="o">+</span><span class="p">(</span><span class="nx">o</span><span class="p">.</span><span class="nx">debug</span><span class="p">?</span><span class="dl">"</span><span class="s2">debug/</span><span class="dl">"</span><span class="p">:</span><span class="dl">""</span><span class="p">))[</span><span class="mi">0</span><span class="p">]</span><span class="o">&&</span><span class="p">(</span><span class="nx">s</span><span class="o">=</span><span class="nx">e</span><span class="p">.</span><span class="nx">querySelectorAll</span><span class="p">(</span><span class="dl">'</span><span class="s1">script[type="text/partytown"]</span><span class="dl">'</span><span class="p">),</span><span class="nx">i</span><span class="o">!=</span><span class="nx">t</span><span class="p">?</span><span class="nx">i</span><span class="p">.</span><span class="nx">dispatchEvent</span><span class="p">(</span><span class="k">new</span> <span class="nx">CustomEvent</span><span class="p">(</span><span class="dl">"</span><span class="s2">pt1</span><span class="dl">"</span><span class="p">,{</span><span class="na">detail</span><span class="p">:</span><span class="nx">t</span><span class="p">})):(</span><span class="nx">d</span><span class="o">=</span><span class="nx">setTimeout</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="mi">1</span><span class="nx">e4</span><span class="p">),</span><span class="nx">e</span><span class="p">.</span><span class="nx">addEventListener</span><span class="p">(</span><span class="dl">"</span><span class="s2">pt0</span><span class="dl">"</span><span class="p">,</span><span class="nx">f</span><span class="p">),</span><span class="nx">r</span><span class="p">?</span><span class="nx">h</span><span class="p">(</span><span class="mi">1</span><span class="p">):</span><span class="nx">n</span><span class="p">.</span><span class="nx">serviceWorker</span><span class="p">?</span><span class="nx">n</span><span class="p">.</span><span class="nx">serviceWorker</span><span class="p">.</span><span class="nx">register</span><span class="p">(</span><span class="nx">a</span><span class="o">+</span><span class="p">(</span><span class="nx">o</span><span class="p">.</span><span class="nx">swPath</span><span class="o">||</span><span class="dl">"</span><span class="s2">partytown-sw.js</span><span class="dl">"</span><span class="p">),{</span><span class="na">scope</span><span class="p">:</span><span class="nx">a</span><span class="p">}).</span><span class="nx">then</span><span class="p">((</span><span class="kd">function</span><span class="p">(</span><span class="nx">t</span><span class="p">){</span><span class="nx">t</span><span class="p">.</span><span class="nx">active</span><span class="p">?</span><span class="nx">h</span><span class="p">():</span><span class="nx">t</span><span class="p">.</span><span class="nx">installing</span><span class="o">&&</span><span class="nx">t</span><span class="p">.</span><span class="nx">installing</span><span class="p">.</span><span class="nx">addEventListener</span><span class="p">(</span><span class="dl">"</span><span class="s2">statechange</span><span class="dl">"</span><span class="p">,(</span><span class="kd">function</span><span class="p">(</span><span class="nx">t</span><span class="p">){</span><span class="dl">"</span><span class="s2">activated</span><span class="dl">"</span><span class="o">==</span><span class="nx">t</span><span class="p">.</span><span class="nx">target</span><span class="p">.</span><span class="nx">state</span><span class="o">&&</span><span class="nx">h</span><span class="p">()}))}),</span><span class="nx">console</span><span class="p">.</span><span class="nx">error</span><span class="p">):</span><span class="nx">w</span><span class="p">())))}</span><span class="kd">function</span> <span class="nx">h</span><span class="p">(</span><span class="nx">t</span><span class="p">){</span><span class="nx">c</span><span class="o">=</span><span class="nx">e</span><span class="p">.</span><span class="nx">createElement</span><span class="p">(</span><span class="nx">t</span><span class="p">?</span><span class="dl">"</span><span class="s2">script</span><span class="dl">"</span><span class="p">:</span><span class="dl">"</span><span class="s2">iframe</span><span class="dl">"</span><span class="p">),</span><span class="nx">t</span><span class="o">||</span><span class="p">(</span><span class="nx">c</span><span class="p">.</span><span class="nx">setAttribute</span><span class="p">(</span><span class="dl">"</span><span class="s2">style</span><span class="dl">"</span><span class="p">,</span><span class="dl">"</span><span class="s2">display:block;width:0;height:0;border:0;visibility:hidden</span><span class="dl">"</span><span class="p">),</span><span class="nx">c</span><span class="p">.</span><span class="nx">setAttribute</span><span class="p">(</span><span class="dl">"</span><span class="s2">aria-hidden</span><span class="dl">"</span><span class="p">,</span><span class="o">!</span><span class="mi">0</span><span class="p">)),</span><span class="nx">c</span><span class="p">.</span><span class="nx">src</span><span class="o">=</span><span class="nx">a</span><span class="o">+</span><span class="dl">"</span><span class="s2">partytown-</span><span class="dl">"</span><span class="o">+</span><span class="p">(</span><span class="nx">t</span><span class="p">?</span><span class="dl">"</span><span class="s2">atomics.js?v=0.7.3</span><span class="dl">"</span><span class="p">:</span><span class="dl">"</span><span class="s2">sandbox-sw.html?</span><span class="dl">"</span><span class="o">+</span><span class="nb">Date</span><span class="p">.</span><span class="nx">now</span><span class="p">()),</span><span class="nx">e</span><span class="p">.</span><span class="nx">body</span><span class="p">.</span><span class="nx">appendChild</span><span class="p">(</span><span class="nx">c</span><span class="p">)}</span><span class="kd">function</span> <span class="nx">w</span><span class="p">(</span><span class="nx">t</span><span class="p">,</span><span class="nx">n</span><span class="p">){</span><span class="k">for</span><span class="p">(</span><span class="nx">f</span><span class="p">(),</span><span class="nx">t</span><span class="o">=</span><span class="mi">0</span><span class="p">;</span><span class="nx">t</span><span class="o"><</span><span class="nx">s</span><span class="p">.</span><span class="nx">length</span><span class="p">;</span><span class="nx">t</span><span class="o">++</span><span class="p">)(</span><span class="nx">n</span><span class="o">=</span><span class="nx">e</span><span class="p">.</span><span class="nx">createElement</span><span class="p">(</span><span class="dl">"</span><span class="s2">script</span><span class="dl">"</span><span class="p">)).</span><span class="nx">innerHTML</span><span class="o">=</span><span class="nx">s</span><span class="p">[</span><span class="nx">t</span><span class="p">].</span><span class="nx">innerHTML</span><span class="p">,</span><span class="nx">e</span><span class="p">.</span><span class="nx">head</span><span class="p">.</span><span class="nx">appendChild</span><span class="p">(</span><span class="nx">n</span><span class="p">);</span><span class="nx">c</span><span class="o">&&</span><span class="nx">c</span><span class="p">.</span><span class="nx">parentNode</span><span class="p">.</span><span class="nx">removeChild</span><span class="p">(</span><span class="nx">c</span><span class="p">)}</span><span class="kd">function</span> <span class="nx">f</span><span class="p">(){</span><span class="nx">clearTimeout</span><span class="p">(</span><span class="nx">d</span><span class="p">)}</span><span class="nx">o</span><span class="o">=</span><span class="nx">t</span><span class="p">.</span><span class="nx">partytown</span><span class="o">||</span><span class="p">{},</span><span class="nx">i</span><span class="o">==</span><span class="nx">t</span><span class="o">&&</span><span class="p">(</span><span class="nx">o</span><span class="p">.</span><span class="nx">forward</span><span class="o">||</span><span class="p">[]).</span><span class="nx">map</span><span class="p">((</span><span class="kd">function</span><span class="p">(</span><span class="nx">e</span><span class="p">){</span><span class="nx">p</span><span class="o">=</span><span class="nx">t</span><span class="p">,</span><span class="nx">e</span><span class="p">.</span><span class="nx">split</span><span class="p">(</span><span class="dl">"</span><span class="s2">.</span><span class="dl">"</span><span class="p">).</span><span class="nx">map</span><span class="p">((</span><span class="kd">function</span><span class="p">(</span><span class="nx">e</span><span class="p">,</span><span class="nx">n</span><span class="p">,</span><span class="nx">i</span><span class="p">){</span><span class="nx">p</span><span class="o">=</span><span class="nx">p</span><span class="p">[</span><span class="nx">i</span><span class="p">[</span><span class="nx">n</span><span class="p">]]</span><span class="o">=</span><span class="nx">n</span><span class="o">+</span><span class="mi">1</span><span class="o"><</span><span class="nx">i</span><span class="p">.</span><span class="nx">length</span><span class="p">?</span><span class="dl">"</span><span class="s2">push</span><span class="dl">"</span><span class="o">==</span><span class="nx">i</span><span class="p">[</span><span class="nx">n</span><span class="o">+</span><span class="mi">1</span><span class="p">]?[]:</span><span class="nx">p</span><span class="p">[</span><span class="nx">i</span><span class="p">[</span><span class="nx">n</span><span class="p">]]</span><span class="o">||</span><span class="p">{}:</span><span class="kd">function</span><span class="p">(){(</span><span class="nx">t</span><span class="p">.</span><span class="nx">_ptf</span><span class="o">=</span><span class="nx">t</span><span class="p">.</span><span class="nx">_ptf</span><span class="o">||</span><span class="p">[]).</span><span class="nx">push</span><span class="p">(</span><span class="nx">i</span><span class="p">,</span><span class="nx">arguments</span><span class="p">)}}))})),</span><span class="dl">"</span><span class="s2">complete</span><span class="dl">"</span><span class="o">==</span><span class="nx">e</span><span class="p">.</span><span class="nx">readyState</span><span class="p">?</span><span class="nx">u</span><span class="p">():(</span><span class="nx">t</span><span class="p">.</span><span class="nx">addEventListener</span><span class="p">(</span><span class="dl">"</span><span class="s2">DOMContentLoaded</span><span class="dl">"</span><span class="p">,</span><span class="nx">u</span><span class="p">),</span><span class="nx">t</span><span class="p">.</span><span class="nx">addEventListener</span><span class="p">(</span><span class="dl">"</span><span class="s2">load</span><span class="dl">"</span><span class="p">,</span><span class="nx">u</span><span class="p">))}(</span><span class="nb">window</span><span class="p">,</span><span class="nb">document</span><span class="p">,</span><span class="nb">navigator</span><span class="p">,</span><span class="nx">top</span><span class="p">,</span><span class="nb">window</span><span class="p">.</span><span class="nx">crossOriginIsolated</span><span class="p">);</span>
<span class="nt"></script></span>
</code></pre></div></div>
<h3 id="test-your-optimizations">Test your optimizations</h3>
<p>Before making any changes, use Chrome developer tools and run a lighthouse scan.</p>
<p>Next identify some potential problematic third-party scripts. Add the <code class="language-plaintext highlighter-rouge">type="text/partytown"</code> attribute to those scripts in your local override. Then re-run the Lighthouse score to observe the impact.</p>
<h2 id="my-findings">My findings</h2>
<p>I tested this on a client staging site today and had very positive results:</p>
<h3 id="initial-metrics">Initial Metrics</h3>
<ul>
<li><strong>TBT</strong> - 2,250ms</li>
<li><strong>Third party blocking time</strong> - 970ms</li>
</ul>
<h3 id="accessibe-snippet-moved-to-partytown">Accessibe snippet moved to Partytown</h3>
<ul>
<li><strong>TBT</strong> - 1,950ms</li>
<li><strong>Third party blocking time</strong> - 640ms</li>
</ul>
<h3 id="accessibe--grin-snippets-moved-to-partytown">Accessibe + Grin snippets moved to Partytown</h3>
<ul>
<li><strong>TBT</strong> - 1,420ms</li>
<li><strong>Third party blocking time</strong> - 680ms (I don’t think Lighthouse knows that Grin is 3rd party)</li>
</ul>
<h3 id="accessibe--grin--powerreviews-moved-to-partytown">Accessibe + Grin + PowerReviews moved to Partytown</h3>
<ul>
<li><strong>TBT</strong> - 620ms</li>
<li><strong>Third party blocking time</strong> - 270ms</li>
</ul>
<h2 id="conclusion">Conclusion</h2>
<p>I’ve just scratched the surface so far with Partytown, but am very excited about the positive performance improvements it can offer for many websites (especially ecommerce). Hope you found this article helpful!</p>maxpchadwickA couple weeks back in a Twitter conversation I learned about Partytown. Would you try to leverage the script off the main thread with Partytown?Didn't use myself yet. Curious about your result.https://t.co/49ifl6QX7I— Denis Metzler (@DenisMetzler) December 13, 2022 I was immediately intrigued by the idea and saw great potential, especially on the ecommerce projects that I work on on a daily basis. Today I spent some time playing with Partytown. In this post I’ll share my process and findings.Magento + OneTrust Cookie Consent - require is not a function2022-12-20T00:00:00+00:002022-12-20T00:00:00+00:00https://maxchadwick.xyz/blog/magento-one-trust-require-is-not-a-function<p>There’s a lot to be said about implementing OneTrust Cookie Consent and this post doesn’t intend to cover it all. Instead, I’d like to share my experience with the JavaScript errors that occured when adding the OneTrust Cookie Consent scripts to a production Magento instance. Here’s a screenshot of the errors from the developer tools:</p>
<p><img class="rounded shadow" src="/img/blog/magento-onetrust-require-is-not-a-function/require-is-not-a-function-dev-tools@1x.png" srcset="/img/blog/magento-onetrust-require-is-not-a-function/require-is-not-a-function-dev-tools@1x.png 1x, /img/blog/magento-onetrust-require-is-not-a-function/require-is-not-a-function-dev-tools@2x.png 2x" alt="Screenshot of errors present in a developer tools" /></p>
<!-- excerpt_separator -->
<p>These errors caused most JavaScript reliant functionality (e.g. add to cart / apply product filters) to break.</p>
<p>In order to resolve this issue I spent quite a bit of time troubleshooting obfuscated code, coordinating with OneTrust support, and reading through OneTrust documentation. In this post I hope to save you some of that time if you’re dealing with the same error.</p>
<h3 id="background-onetrust-auto-blocking">Background: OneTrust Auto-Blocking</h3>
<p>If you’re dealing with this error on a Magento site, like me, there’s a good chance you’re using <a href="https://my.onetrust.com/articles/en_US/Knowledge/UUID-c5122557-2070-65cb-2612-f2752c0cc4aa">OneTrust Auto-Block</a>. Using the Auto-Blocking technology makes a lot of sense because without it, there’s a non-trivial engineering lift to ensure that cookies are actually blocked (see: <a href="https://community.cookiepro.com/s/article/Cookie-Blocking-Blocking-cookies-via-Tag-Managers-and-HTML?language=en_US">OneTrust: Cookie Blocking - Blocking Cookies via Tag Managers and HTML Implementation Webinar</a>). Using auto-block in theory simplifies this as we can just have OneTrust automatically block scripts without needing to change anything within the website markup or tag manager configuration.</p>
<h3 id="tracing-the-error-to-the-auto-blocking-script">Tracing the error to the auto-blocking script</h3>
<p>When this error occured I confirmed it was specifically the auto-blocking script (e.g. https://cdn.cookielaw.org/consent/SITE-ID/OtAutoBlock.js) that was causing this.
Confirming this fairly easy. Using Chrome developer tools <a href="https://developer.chrome.com/blog/new-in-devtools-65/#overrides">local overrides</a> I tried removing just the auto-block script, but leaving the otSDKStub.js script, and observed that the error was no longer happening.</p>
<h3 id="digging-in-to-the-auto-block-script">Digging in to the auto-block script</h3>
<p>The next question became what part of that script was causing the error. This was time consuming and difficult to do (the fact that the file is obfuscated doesn’t help with this). One of the things in the auto-block script that stood out to me was as a variable that looked something like this…</p>
<div class="language-plaintext wrap highlighter-rouge"><div class="highlight"><pre class="highlight"><code>var x = JSON.parse('[{"Tag":"https://www.googletagmanager.com/gtm.js","CategoryId":["C0002"],"Vendor":null},{"Tag":"https://dpm.demdex.net/ibs:dpid\x3d28645\x26dpuuid\x3d","CategoryId":["C0004"],"Vendor":null},{"Tag":"https://contextual.media.net/cksync.php","CategoryId":["C0004"],"Vendor":null},
</code></pre></div></div>
<p>With some clues from OneTrust I realized that this variable contains a mapping of “Tag”s (which are just URLs) to OneTrust cookie categories (e.g. C0002).</p>
<p>I was eventually able to determine that auto-block script checks each script as it’s added to the page against this list of tags (using a <code class="language-plaintext highlighter-rouge">MutationObserver</code>). If the script <code class="language-plaintext highlighter-rouge">src</code> is contained in the list of tags it switches the <code class="language-plaintext highlighter-rouge">type</code> from <code class="language-plaintext highlighter-rouge">application/javascript</code> to <code class="language-plaintext highlighter-rouge">text/plain</code> and adds a class attribute listing with the appropriate cookie categories. This will prevent the script from being executed.</p>
<p>Later, in the <code class="language-plaintext highlighter-rouge">otBannerSdk.js</code> JavaScript the <code class="language-plaintext highlighter-rouge">reactivateScriptTag</code> function will run, and it will replace the <code class="language-plaintext highlighter-rouge">text/plain</code> script tags with new script tags with type <code class="language-plaintext highlighter-rouge">application/javascript</code>.</p>
<h3 id="so-why-were-we-getting-the-error">So why were we getting the error</h3>
<p>What I was eventually able to determine was that the tag list variable for this site contained a tag like this</p>
<div class="language-plaintext wrap highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{"Tag":"https://www.example.com/","CategoryId":["C0002"],"Vendor":null}
</code></pre></div></div>
<p>I’ve replaced the actual client’s domain with example.com here. As you can see somehow the home page of the website was listed as a “Tag”.</p>
<p>The effect of this was that script tags such as the <code class="language-plaintext highlighter-rouge">require.js</code> tag were actually subject to the auto-block logic. They were getting a <code class="language-plaintext highlighter-rouge">text/plain</code> type applied initially to them on page load. Later the <code class="language-plaintext highlighter-rouge">reactivateScriptTag</code> would re-enable them (assuming the user hadn’t opt-ed out of that cookie category – we were testing with an Opt-Out consent model). However before <code class="language-plaintext highlighter-rouge">reactivateScriptTag</code> would run there were other inline scripts that were trying to call <code class="language-plaintext highlighter-rouge">require</code> or <code class="language-plaintext highlighter-rouge">require.config</code>, hence the error.</p>
<h3 id="the-root-of-the-issue">The root of the issue</h3>
<p>The remaining open question was, how does the tag list variable get generated? Through some hints from OneTrust support what I determined was that the tag configuration in the auto-block script is actually generated based on the “sources” attributed to each cookie.</p>
<p><img class="rounded shadow" src="/img/blog/magento-onetrust-require-is-not-a-function/onetrust-cookie-sources@1x.png" srcset="/img/blog/magento-onetrust-require-is-not-a-function/onetrust-cookie-sources@1x.png 1x, /img/blog/magento-onetrust-require-is-not-a-function/onetrust-cookie-sources@2x.png 2x" alt="Screenshot of cookie sources in OneTrust UI" /></p>
<p>OneTrust attempts to determine the cookie sources during the site scan, but in our case, many cookies had actually been incorrectly attributed to the client’s home page (e.g. https://www.example.com). While I’m not sure why the scan had this result (interestingly this didn’t happen in lower environments), OneTrust actually pointed me to some of their documentation which notes that when using auto-block it’s necessary to manually review the cookie sources for all the cookies to make sure it’s correct.</p>
<h3 id="fixing-the-issue">Fixing the issue</h3>
<p>In order the fix the issue I had to review the source for each and every cookie one-by-one and determine the correct sources. For non-Magento issued cookies clearly it shouldn’t be a URL under example.com. For the Magento cookies, the best seems to be just remove any source (<a href="https://docs.magento.com/user-guide/v2.3/stores/cookie-reference.html">Magento documentation</a> states that their cookies are exempt). Generally I was able to correct the sources through the OneTrust recommended workflow, but in some cases cookies had been incorrectly attributed to nearly 2,000 URLs on the client’s site. For these cases the solution I came up with was the delete and manually re-create the cookie in the UI.</p>
<h3 id="conclusion">Conclusion</h3>
<p>In the end implementing cookie blocking is a lot of work even if you use the auto-block. That being said, I still believe that using auto-block will save a lot of engineering time compared to manual blocking and continue to think it’s the best option when implementing OneTrust cookie consent.</p>maxpchadwickThere’s a lot to be said about implementing OneTrust Cookie Consent and this post doesn’t intend to cover it all. Instead, I’d like to share my experience with the JavaScript errors that occured when adding the OneTrust Cookie Consent scripts to a production Magento instance. Here’s a screenshot of the errors from the developer tools:How Magento’s JavaScript Block Loader Works2022-12-11T00:00:00+00:002022-12-11T00:00:00+00:00https://maxchadwick.xyz/blog/how-magento-javascript-block-loader-works<p>I’ve been looking at a Magento site where multiple loaders show up in different areas on the cart page. The loaders look something like this:</p>
<p><img class="rounded shadow" src="/img/blog/magento-javascript-block-loader/block-loader.jpeg" alt="Screenshot of what the loader looks like" /></p>
<p>I did a bit of a deep dive into what actually causes these loader to show up. In this post I’ll share my findings.</p>
<!-- excerpt_separator -->
<h3 id="magento_uijsblock-loader">Magento_Ui/js/block-loader</h3>
<p>Loaders that show up in a specific area of the page are typically powered by <code class="language-plaintext highlighter-rouge">Magento_Ui/js/block-loader</code>.</p>
<p>This component creates a <a href="https://knockoutjs.com/documentation/custom-bindings.html">custom knockout binding</a> called <code class="language-plaintext highlighter-rouge">blockLoader</code></p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">ko</span><span class="p">.</span><span class="nx">bindingHandlers</span><span class="p">.</span><span class="nx">blockLoader</span> <span class="o">=</span> <span class="p">{</span>
<span class="cm">/**
* Process loader for block
* @param {String} element
* @param {Boolean} displayBlockLoader
*/</span>
<span class="na">update</span><span class="p">:</span> <span class="kd">function</span> <span class="p">(</span><span class="nx">element</span><span class="p">,</span> <span class="nx">displayBlockLoader</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">element</span> <span class="o">=</span> <span class="nx">$</span><span class="p">(</span><span class="nx">element</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">ko</span><span class="p">.</span><span class="nx">unwrap</span><span class="p">(</span><span class="nx">displayBlockLoader</span><span class="p">()))</span> <span class="p">{</span>
<span class="nx">blockLoaderElement</span><span class="p">.</span><span class="nx">done</span><span class="p">(</span><span class="nx">addBlockLoader</span><span class="p">(</span><span class="nx">element</span><span class="p">));</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="nx">blockLoaderElement</span><span class="p">.</span><span class="nx">done</span><span class="p">(</span><span class="nx">removeBlockLoader</span><span class="p">(</span><span class="nx">element</span><span class="p">));</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">};</span>
</code></pre></div></div>
<p>Reference: <a href="https://github.com/magento/magento2/blob/2.4.5-p1/app/code/Magento/Ui/view/base/web/js/block-loader.js#L76-L93">https://github.com/magento/magento2/blob/2.4.5-p1/app/code/Magento/Ui/view/base/web/js/block-loader.js#L76-L93</a></p>
<p>If the binding value is <code class="language-plaintext highlighter-rouge">true</code> the <code class="language-plaintext highlighter-rouge">addBlockLoader</code> function will be called (and if <code class="language-plaintext highlighter-rouge">false</code> it will call <code class="language-plaintext highlighter-rouge">removeBlockLoader</code>.</p>
<h3 id="using-the-binding">Using the binding</h3>
<p><code class="language-plaintext highlighter-rouge">Magento_Checkout/js/view/cart/totals</code> provides a good example of how to use the binding.</p>
<p>The component has an <code class="language-plaintext highlighter-rouge">isLoading</code> property, which is mapped to <code class="language-plaintext highlighter-rouge">isLoading</code> in <code class="language-plaintext highlighter-rouge">Magento_Checkout/js/model/totals</code></p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">define</span><span class="p">([</span>
<span class="dl">'</span><span class="s1">jquery</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">uiComponent</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">Magento_Checkout/js/model/totals</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">Magento_Checkout/js/model/shipping-service</span><span class="dl">'</span>
<span class="p">],</span> <span class="kd">function</span> <span class="p">(</span><span class="nx">$</span><span class="p">,</span> <span class="nx">Component</span><span class="p">,</span> <span class="nx">totalsService</span><span class="p">,</span> <span class="nx">shippingService</span><span class="p">)</span> <span class="p">{</span>
<span class="dl">'</span><span class="s1">use strict</span><span class="dl">'</span><span class="p">;</span>
<span class="k">return</span> <span class="nx">Component</span><span class="p">.</span><span class="nx">extend</span><span class="p">({</span>
<span class="na">isLoading</span><span class="p">:</span> <span class="nx">totalsService</span><span class="p">.</span><span class="nx">isLoading</span><span class="p">,</span>
</code></pre></div></div>
<p>Reference: <a href="https://github.com/magento/magento2/blob/2.4.5-p1/app/code/Magento/Checkout/view/frontend/web/js/view/cart/totals.js#L5-L14">https://github.com/magento/magento2/blob/2.4.5-p1/app/code/Magento/Checkout/view/frontend/web/js/view/cart/totals.js#L5-L14</a></p>
<p>Then in the template file <code class="language-plaintext highlighter-rouge">Magento_Checkout/template/cart/totals.html</code> we can see the custom binding set on the wrapper div.</p>
<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><div</span> <span class="na">class=</span><span class="s">"table-wrapper"</span> <span class="na">data-bind=</span><span class="s">"blockLoader: isLoading"</span><span class="nt">></span>
</code></pre></div></div>
<p>Reference: <a href="https://github.com/magento/magento2/blob/2.4.5-p1/app/code/Magento/Checkout/view/frontend/web/template/cart/totals.html">https://github.com/magento/magento2/blob/2.4.5-p1/app/code/Magento/Checkout/view/frontend/web/template/cart/totals.html</a></p>
<p>In <code class="language-plaintext highlighter-rouge">Magento_Checkout/js/model/totals</code> we can see that <code class="language-plaintext highlighter-rouge">isLoading</code> is an observable, initially set to <code class="language-plaintext highlighter-rouge">false</code>.</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">isLoading</span><span class="p">:</span> <span class="nx">ko</span><span class="p">.</span><span class="nx">observable</span><span class="p">(</span><span class="kc">false</span><span class="p">),</span>
</code></pre></div></div>
<p>Reference: <a href="https://github.com/magento/magento2/blob/2.4.5-p1/app/code/Magento/Checkout/view/frontend/web/js/model/totals.js#L31">https://github.com/magento/magento2/blob/2.4.5-p1/app/code/Magento/Checkout/view/frontend/web/js/model/totals.js#L31</a></p>
<p>If we search for <code class="language-plaintext highlighter-rouge">isLoading</code> across the codebase we find many places that toggle the value, which will cause the loader to show. For example in <code class="language-plaintext highlighter-rouge">Magento_Checkout/js/action/get-totals.js</code></p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nx">totals</span><span class="p">.</span><span class="nx">isLoading</span><span class="p">(</span><span class="kc">true</span><span class="p">);</span>
<span class="k">return</span> <span class="nx">storage</span><span class="p">.</span><span class="kd">get</span><span class="p">(</span>
<span class="nx">resourceUrlManager</span><span class="p">.</span><span class="nx">getUrlForCartTotals</span><span class="p">(</span><span class="nx">quote</span><span class="p">),</span>
<span class="kc">false</span>
<span class="p">).</span><span class="nx">done</span><span class="p">(</span><span class="kd">function</span> <span class="p">(</span><span class="nx">response</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">proceed</span> <span class="o">=</span> <span class="kc">true</span><span class="p">;</span>
<span class="nx">totals</span><span class="p">.</span><span class="nx">isLoading</span><span class="p">(</span><span class="kc">false</span><span class="p">);</span>
</code></pre></div></div>
<p>Reference: <a href="https://github.com/magento/magento2/blob/2.4.5-p1/app/code/Magento/Checkout/view/frontend/web/js/action/get-totals.js#L21-L29">https://github.com/magento/magento2/blob/2.4.5-p1/app/code/Magento/Checkout/view/frontend/web/js/action/get-totals.js#L21-L29</a></p>
<h3 id="monitoring-loader-display">Monitoring loader display</h3>
<p>One thing that could be interesting to put in place is monitoring to track how much time the site users spend seeing loaders. Doing so would require patching <code class="language-plaintext highlighter-rouge">Magento_Ui/js/block-loader</code>.</p>
<p>Adding a tracker that flags when the block loader shows and when it’s hidden can be done as follows.</p>
<p><strong><code class="language-plaintext highlighter-rouge">addBlockLoader</code></strong></p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">function</span> <span class="nx">addBlockLoader</span><span class="p">(</span><span class="nx">element</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s2">`addBlockLoader </span><span class="p">${</span><span class="nx">element</span><span class="p">.</span><span class="nx">context</span><span class="p">.</span><span class="nx">className</span><span class="p">}</span><span class="s2"> </span><span class="p">${</span><span class="nx">element</span><span class="p">.</span><span class="nx">context</span><span class="p">.</span><span class="nx">id</span><span class="p">}</span><span class="s2"> </span><span class="p">${</span><span class="nx">performance</span><span class="p">.</span><span class="nx">now</span><span class="p">()}</span><span class="s2">`</span><span class="p">);</span>
</code></pre></div></div>
<p><strong><code class="language-plaintext highlighter-rouge">removeBlockLoader</code></strong></p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">function</span> <span class="nx">removeBlockLoader</span><span class="p">(</span><span class="nx">element</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nx">element</span><span class="p">.</span><span class="nx">has</span><span class="p">(</span><span class="nx">blockLoaderClass</span><span class="p">).</span><span class="nx">length</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span><span class="p">;</span>
<span class="p">}</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s2">`removeBlockLoader </span><span class="p">${</span><span class="nx">element</span><span class="p">.</span><span class="nx">context</span><span class="p">.</span><span class="nx">className</span><span class="p">}</span><span class="s2"> </span><span class="p">${</span><span class="nx">element</span><span class="p">.</span><span class="nx">context</span><span class="p">.</span><span class="nx">id</span><span class="p">}</span><span class="s2"> </span><span class="p">${</span><span class="nx">performance</span><span class="p">.</span><span class="nx">now</span><span class="p">()}</span><span class="s2">`</span><span class="p">);</span>
</code></pre></div></div>
<p>From here we can extrapolate how much time the loader was visibile for, and on which area of the page.</p>
<p>Obviously in the real world we would not use <code class="language-plaintext highlighter-rouge">console.log</code> and instead something like <a href="https://docs.newrelic.com/docs/browser/new-relic-browser/browser-apis/addpageaction/"><code class="language-plaintext highlighter-rouge">newrelic.addPageAction</code></a> (assuming the project is using New Relic).</p>
<h3 id="wrap-up">Wrap Up</h3>
<p>That’s all for now. Hopefully you found this helpful because I sure as heck was scratching my head trying to figure out what was causing the spinners to show up.</p>maxpchadwickI’ve been looking at a Magento site where multiple loaders show up in different areas on the cart page. The loaders look something like this: I did a bit of a deep dive into what actually causes these loader to show up. In this post I’ll share my findings.Getting the Current Fastly VCL via API2022-09-28T00:00:00+00:002022-09-28T00:00:00+00:00https://maxchadwick.xyz/blog/getting-the-current-fastly-vcl-via-api<p>This is just a quick little tip, but something I always have to look up how to do. The Fastly API has <a href="https://developer.fastly.com/reference/api/vcl-services/vcl/#get-custom-vcl-generated">an endpoint for fetching the generated VCL for a service</a>. The issue is that the <code class="language-plaintext highlighter-rouge">version_id</code> for the serivce must be provided in the request (there’s no way to say “just give me the currently active VCL”).</p>
<!-- excerpt_separator -->
<p>As such, we need to first use the <a href="https://developer.fastly.com/reference/api/services/version/#list-service-versions">List versions of a service</a> endpoint to identify the active version. From there we can request the generated VCL.</p>
<p>Putting this all together, assuming you have <a href="https://stedolan.github.io/jq/"><code class="language-plaintext highlighter-rouge">jq</code></a> installed, here’s how to do this in practice (replace <code class="language-plaintext highlighter-rouge">FASTLY_KEY</code> and <code class="language-plaintext highlighter-rouge">SERVICE_ID</code> with the correct values for your project).</p>
<div class="tout tout--secondary">
<p>For Magento Cloud environments, the Fastly Key and Service ID can be found in the <code>/mnt/shared/fastly_tokens.txt</code> file. "API Token" is the <code>FASTLY_KEY</code> and "Serivce ID" is the <code>SERVICE_ID</code>.</p>
</div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># Get the active version. In this example 105 is active
$ curl --silent -H "Fastly-Key: FASTLY_KEY" https://api.fastly.com/service/SERVICE_ID/version \
| jq '.[] | if .active then .number else empty end'
105
# Get the VCL
$ curl -H "Fastly-Key: FASTLY_KEY" https://api.fastly.com/service/SERVICE_ID/version/105/generated_vcl
{"version":105,"service_id":"...
</code></pre></div></div>maxpchadwickThis is just a quick little tip, but something I always have to look up how to do. The Fastly API has an endpoint for fetching the generated VCL for a service. The issue is that the version_id for the serivce must be provided in the request (there’s no way to say “just give me the currently active VCL”).Preventing Flag Conflicts in Go2022-07-18T00:00:00+00:002022-07-18T00:00:00+00:00https://maxchadwick.xyz/blog/preventing-flag-conflicts-in-go<p>After <code class="language-plaintext highlighter-rouge">import</code>-ing a new package into one of my go projects and attempting to run the build, I was presented the following error:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>panic: flag redefined: version
</code></pre></div></div>
<p>In my project, the <code class="language-plaintext highlighter-rouge">version</code> flag allows the user to see what version of the tool they have installed (<code class="language-plaintext highlighter-rouge">main.version</code> is passed as the current git tag via <code class="language-plaintext highlighter-rouge">-ldflags</code> in the build script).</p>
<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">package</span> <span class="n">main</span>
<span class="k">import</span> <span class="p">(</span>
<span class="s">"flag"</span>
<span class="s">"fmt"</span>
<span class="p">)</span>
<span class="k">var</span> <span class="n">version</span> <span class="kt">string</span>
<span class="k">func</span> <span class="n">main</span><span class="p">()</span> <span class="p">{</span>
<span class="n">ver</span> <span class="o">:=</span> <span class="n">flag</span><span class="o">.</span><span class="n">Bool</span><span class="p">(</span><span class="s">"version"</span><span class="p">,</span> <span class="no">false</span><span class="p">,</span> <span class="s">"Get current version"</span><span class="p">)</span>
<span class="n">flag</span><span class="o">.</span><span class="n">Parse</span><span class="p">()</span>
<span class="k">if</span> <span class="o">*</span><span class="n">ver</span> <span class="p">{</span>
<span class="n">fmt</span><span class="o">.</span><span class="n">Println</span><span class="p">(</span><span class="n">ver</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Presumably, the problem was that the newly imported package also used a flag with the same name.</p>
<!-- excerpt_separator -->
<p>The first solution that came to mind was to rename my flag from <code class="language-plaintext highlighter-rouge">version</code> to something else (e.g. <code class="language-plaintext highlighter-rouge">ver</code>). While this might be a viable approach to fix the issue it was definitely not a desirable one. As such, I <a href="https://github.com/vitessio/vitess/issues/10714">reported the issue</a> to the package provider to see if they’d be willing to make any compatibility adjustments on their end to accommodate my use case.</p>
<p>While initially it appeared that my only recourse would be to rename the flag, a clever user <a href="https://github.com/vitessio/vitess/issues/10714#issuecomment-1186579881">responded</a> to my issue, making me aware of the <code class="language-plaintext highlighter-rouge">flag.NewFlagSet</code> function. By calling the <code class="language-plaintext highlighter-rouge">NewFlagSet</code> function in my code before defining my flags I was able to prevent conflicts with the newly imported package.</p>
<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">package</span> <span class="n">main</span>
<span class="k">import</span> <span class="p">(</span>
<span class="s">"flag"</span>
<span class="s">"fmt"</span>
<span class="p">)</span>
<span class="k">var</span> <span class="n">version</span> <span class="kt">string</span>
<span class="k">func</span> <span class="n">main</span><span class="p">()</span> <span class="p">{</span>
<span class="n">flag</span><span class="o">.</span><span class="n">CommandLine</span> <span class="o">=</span> <span class="n">flag</span><span class="o">.</span><span class="n">NewFlagSet</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">Args</span><span class="p">[</span><span class="m">0</span><span class="p">],</span> <span class="n">flag</span><span class="o">.</span><span class="n">ExitOnError</span><span class="p">)</span>
<span class="n">ver</span> <span class="o">:=</span> <span class="n">flag</span><span class="o">.</span><span class="n">Bool</span><span class="p">(</span><span class="s">"version"</span><span class="p">,</span> <span class="no">false</span><span class="p">,</span> <span class="s">"Get current version"</span><span class="p">)</span>
<span class="n">flag</span><span class="o">.</span><span class="n">Parse</span><span class="p">()</span>
<span class="k">if</span> <span class="o">*</span><span class="n">ver</span> <span class="p">{</span>
<span class="n">fmt</span><span class="o">.</span><span class="n">Println</span><span class="p">(</span><span class="n">ver</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>maxpchadwickAfter import-ing a new package into one of my go projects and attempting to run the build, I was presented the following error: panic: flag redefined: version In my project, the version flag allows the user to see what version of the tool they have installed (main.version is passed as the current git tag via -ldflags in the build script). package main import ( "flag" "fmt" ) var version string func main() { ver := flag.Bool("version", false, "Get current version") flag.Parse() if *ver { fmt.Println(ver) } } Presumably, the problem was that the newly imported package also used a flag with the same name.Troubleshooting mismatched anonymous define2021-07-09T00:00:00+00:002021-07-09T00:00:00+00:00https://maxchadwick.xyz/blog/troubleshooting-mismatched-anonymous-define<p>If you’re reading this post you’re probably troubleshooting an error like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(index):10 Uncaught Error: Mismatched anonymous define() module: function(){return wr}
http://requirejs.org/docs/errors.html#mismatch
at makeError (require.min.js:formatted:86)
at intakeDefines (require.min.js:formatted:713)
at require.min.js:formatted:835
at nrWrapper ((index):10)
</code></pre></div></div>
<p>While the RequireJS documentation provides <a href="https://requirejs.org/docs/errors.html#mismatch">some direction on this error</a>, in practice it can be a nightmare to understand what’s actually happening as the error trace provides no indication as to the underlying code causing the issue.</p>
<p>Fortunately, my colleague found an approach for getting to the actual source of the problem. In this post I’ll share that approach.</p>
<!-- excerpt_separator -->
<h3 id="tracing-back-the-error">Tracing Back The Error</h3>
<p>The key to tracing the error is to set a breakpoint at the place where <code class="language-plaintext highlighter-rouge">makeError</code> is called. In the trace above that would be inside the <code class="language-plaintext highlighter-rouge">intakeDefines</code> function.</p>
<p><img class="rounded shadow" src="/img/blog/uncaught-mismatched-define/dev-tools@1x.png" srcset="/img/blog/uncaught-mismatched-define/dev-tools@1x.png 1x, /img/blog/uncaught-mismatched-define/dev-tools@2x.png 2x" alt="Screenshot location in intakeDefines to set breakpoint" /></p>
<p>Once the breakpoint is set, trigger the error (typically just refresh the page).</p>
<p>When it hits the breakpoint, look at the <code class="language-plaintext highlighter-rouge">args</code> variable. Specifically look for the <code class="language-plaintext highlighter-rouge">[[FunctionLocation]]</code> symbol key. This should contain the URL of the script that triggered the error.</p>
<p><img class="rounded shadow" src="/img/blog/uncaught-mismatched-define/function-location@1x.png" srcset="/img/blog/uncaught-mismatched-define/function-location@1x.png 1x, /img/blog/uncaught-mismatched-define/function-location@2x.png 2x" alt="Finding the URL that triggered the error via FunctionLocation" /></p>
<p>Once you’ve identified the script you’ll have to decide what to do. In all the cases I’ve seen of this it’s been caused by third party JavaScript. In the example above the problematic code looked like this:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">e</span><span class="p">.</span><span class="nx">prototype</span><span class="p">[</span><span class="dl">"</span><span class="s2">catch</span><span class="dl">"</span><span class="p">]</span> <span class="o">=</span> <span class="kc">null</span><span class="p">;</span>
<span class="kd">var</span> <span class="nx">c</span> <span class="o">=</span> <span class="nx">t</span><span class="p">.</span><span class="nx">fastdom</span> <span class="o">=</span> <span class="nx">t</span><span class="p">.</span><span class="nx">fastdom</span> <span class="o">||</span> <span class="k">new</span> <span class="nx">e</span><span class="p">;</span>
<span class="dl">"</span><span class="s2">f</span><span class="dl">"</span> <span class="o">==</span> <span class="p">(</span><span class="k">typeof</span> <span class="nx">define</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span> <span class="p">?</span> <span class="nx">define</span><span class="p">(</span><span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">c</span>
<span class="p">})</span> <span class="p">:</span> <span class="dl">"</span><span class="s2">o</span><span class="dl">"</span> <span class="o">==</span> <span class="p">(</span><span class="k">typeof</span> <span class="nx">module</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span> <span class="o">&&</span> <span class="p">(</span><span class="nx">module</span><span class="p">.</span><span class="nx">exports</span> <span class="o">=</span> <span class="nx">c</span><span class="p">)</span>
<span class="p">}(</span><span class="dl">"</span><span class="s2">undefined</span><span class="dl">"</span> <span class="o">!=</span> <span class="k">typeof</span> <span class="nb">window</span> <span class="p">?</span> <span class="nb">window</span> <span class="p">:</span> <span class="k">this</span><span class="p">),</span>
</code></pre></div></div>
<p>Specifically calling <code class="language-plaintext highlighter-rouge">define</code> like this is no bueno with RequireJS<sup style="display: inline-block" id="a1"><a href="#f1">1</a></sup>.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>define(function() {
</code></pre></div></div>
<p>If it’s a third party, likely the best option in the immediate term is to remove the script from the site. Then report the issue to the provider - specifically that their script is not compatible with RequireJS (and as such not compatible with Magento 2, if you’re using that). The script can be added back to the site once the provider remediates the issue (assuming they are willing to do so).</p>
<h3 id="footnotes">Footnotes</h3>
<p><b id="f1">1 </b>. This is a slight over-simplification, but the bottom line is that the third party script in it’s entirety isn’t compatible with RequireJS<a href="#a1">↩</a></p>maxpchadwickIf you’re reading this post you’re probably troubleshooting an error like this: (index):10 Uncaught Error: Mismatched anonymous define() module: function(){return wr} http://requirejs.org/docs/errors.html#mismatch at makeError (require.min.js:formatted:86) at intakeDefines (require.min.js:formatted:713) at require.min.js:formatted:835 at nrWrapper ((index):10) While the RequireJS documentation provides some direction on this error, in practice it can be a nightmare to understand what’s actually happening as the error trace provides no indication as to the underlying code causing the issue. Fortunately, my colleague found an approach for getting to the actual source of the problem. In this post I’ll share that approach.Finding Largest Tables in MySQL 8 / MariaDB 10.22021-01-08T00:00:00+00:002021-01-08T00:00:00+00:00https://maxchadwick.xyz/blog/finding-largest-tables-mysql-8-mariadb-10-2<p>If you’re like me the Percona blog’s <a href="https://www.percona.com/blog/2008/02/04/finding-out-largest-tables-on-mysql-server/">“Finding the largest tables on MySQL Server”</a> from 2008 is a resource you frequently visit.</p>
<p>However, when running the query recently I experienced the following error:</p>
<div class="language-plaintext wrap highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'rows,
</code></pre></div></div>
<!-- excerpt_separator -->
<p>Playing around a little bit I determined that the issue was caused by the fact that the query in the Percona article uses <code class="language-plaintext highlighter-rouge">rows</code> as an identifier name without quoting.</p>
<p>It turns out that <a href="https://dev.mysql.com/doc/refman/8.0/en/keywords.html#keywords-8-0-detailed-R">as of MySQL version 8.0.2</a> and <a href="https://mariadb.com/kb/en/mariadb-1024-release-notes/">MariaDB version 10.2.4</a> <code class="language-plaintext highlighter-rouge">rows</code> is now a reserved word and cannot be used as an identifier name without being quoted.</p>
<p>All that’s needed to resolve the issue is to quote rows like this: <code class="language-plaintext highlighter-rouge">`rows`</code>.</p>
<p>Here’s the updated full query that’s compatible with MySQL 8 / MariaDB 10.2:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>SELECT CONCAT(table_schema, '.', table_name),
CONCAT(ROUND(table_rows / 1000000, 2), 'M') `rows`,
CONCAT(ROUND(data_length / ( 1024 * 1024 * 1024 ), 2), 'G') DATA,
CONCAT(ROUND(index_length / ( 1024 * 1024 * 1024 ), 2), 'G') idx,
CONCAT(ROUND(( data_length + index_length ) / ( 1024 * 1024 * 1024 ), 2), 'G') total_size,
ROUND(index_length / data_length, 2) idxfrac
FROM information_schema.TABLES
ORDER BY data_length + index_length DESC
LIMIT 10;
</code></pre></div></div>maxpchadwickIf you’re like me the Percona blog’s “Finding the largest tables on MySQL Server” from 2008 is a resource you frequently visit. However, when running the query recently I experienced the following error: ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'rows,Fastly Timeouts Caused By Request Collapsing2021-01-07T00:00:00+00:002021-01-07T00:00:00+00:00https://maxchadwick.xyz/blog/fastly-timeouts-caused-by-request-collapsing<p>Recently I was involved in diagnosing an odd issue. For a specific page on a website, customers were experiencing extremely slow responses, frequently timing out with the following message.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Timed out while waiting on cache-dca17735-DCA
</code></pre></div></div>
<div class="tout tout--secondary">
<p><strong>NOTE</strong>: <code>cache-dca17735-DCA</code> in this case refers to Fastly's DCA data center in Ashburn, VA. (See: <a href="https://www.fastly.com/release-notes/q2-17-q3-17">https://www.fastly.com/release-notes/q2-17-q3-17</a>). This aspect of the response would change from user to user, however the "Timed out while waiting on" part was consistent.</p>
</div>
<p>However, when we looked at the application backend we saw no sign that it was unhealthy in any way. While we did see that throughput was slightly elevated for the route in question, average server response times were quick, and there were no signs that any of the infrastructure was overloaded.</p>
<!-- excerpt_separator -->
<p>We reported the issue to Fastly support, who advised that it was likely caused <a href="https://developer.fastly.com/learning/concepts/request-collapsing/">Request collapsing</a>. I’m not 100% sure of the conditions that can lead to this happening but, the following were at play for us:</p>
<ul>
<li>Throughput to the problematic URL was slightly elevated</li>
<li>The URL would return a 307 response code, for which Fastly supported advised <code class="language-plaintext highlighter-rouge">beresp.cacheable</code> would be set to <code class="language-plaintext highlighter-rouge">false</code> (Reference: <a href="https://docs.fastly.com/en/guides/http-status-codes-cached-by-default">https://docs.fastly.com/en/guides/http-status-codes-cached-by-default</a>).</li>
</ul>
<p>Apparently request collapsing creates a queue for requests, which can become backed up if a specific URL gets enough traffic under the right conditions.</p>
<p>The fix they suggested was to add a <code class="language-plaintext highlighter-rouge">return (pass)</code> targeting this specific URL to the <code class="language-plaintext highlighter-rouge">vcl_recv</code> subroutine.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sub vcl_recv {
if (req.url.path ~ "/problem-url") {
return(pass);
}
}
</code></pre></div></div>
<div class="tout tout--secondary">
<p><strong>NOTE</strong>: For Magento users this can be done via a <a href="https://github.com/fastly/fastly-magento2/blob/1.2.152/Documentation/Guides/CUSTOM-VCL-SNIPPETS.md">Custom VCL snippet</a></p>
</div>
<p>Upon implementing this change we found that the issue no longer occured.</p>maxpchadwickRecently I was involved in diagnosing an odd issue. For a specific page on a website, customers were experiencing extremely slow responses, frequently timing out with the following message. Timed out while waiting on cache-dca17735-DCA NOTE: cache-dca17735-DCA in this case refers to Fastly's DCA data center in Ashburn, VA. (See: https://www.fastly.com/release-notes/q2-17-q3-17). This aspect of the response would change from user to user, however the "Timed out while waiting on" part was consistent. However, when we looked at the application backend we saw no sign that it was unhealthy in any way. While we did see that throughput was slightly elevated for the route in question, average server response times were quick, and there were no signs that any of the infrastructure was overloaded.