Blog
grep color Adding ANSI Escape Sequence
Published: September 19, 2018
Today I was using some command line kung-fu to transform a CSV with thousands of URL redirects into a text file containing the same redirects in Apache mod_rewrite format.
The command looked roughly like this 1…
$ cat redirects-with-store-ids.csv | grep ',17' | awk -F"," '{ print "redirect 301 /" $1 " https://www.example.com/" $2 }' > example-com-redirects.conf
After getting the command just right I opened generated the file in Sublime Text and saw some strange looking characters at the end of each line…
redirect 301 /page/beige.html https://www.example.com/page.html?color=51<0x1b>[01;31m<0x1b>[K
redirect 301 /page/black.html https://www.example.com/page.html?color=52<0x1b>[01;31m<0x1b>[K
redirect 301 /page/blue.html https://www.example.com/page.html?color=53<0x1b>[01;31m<0x1b>[K
redirect 301 /page/brown.html https://www.example.com/page.html?color=55<0x1b>[01;31m<0x1b>[K
redirect 301 /page/green.html https://www.example.com/page.html?color=57<0x1b>[01;31m<0x1b>[K
For some reason “<0x1b>[01;31m<0x1b>[K” had been appended at the end of every line.
Customer Grid Index Doesn't Update on Schedule
Published: September 17, 2018
As a performance and scalability optimization, it is recommended to set all of Magento’s indexers to ‘Update by Schedule’ mode. This can be done at the command line as follows…
$ php bin/magento indexer:set-mode schedule
Index mode for Indexer Design Config Grid was changed from 'Update on Save' to 'Update by Schedule'
Index mode for Indexer Customer Grid was changed from 'Update on Save' to 'Update by Schedule'
Index mode for Indexer Category Products was changed from 'Update on Save' to 'Update by Schedule'
Index mode for Indexer Product Categories was changed from 'Update on Save' to 'Update by Schedule'
Index mode for Indexer Product Price was changed from 'Update on Save' to 'Update by Schedule'
Index mode for Indexer Product EAV was changed from 'Update on Save' to 'Update by Schedule'
Index mode for Indexer Stock was changed from 'Update on Save' to 'Update by Schedule'
Index mode for Indexer Catalog Rule Product was changed from 'Update on Save' to 'Update by Schedule'
Index mode for Indexer Catalog Product Rule was changed from 'Update on Save' to 'Update by Schedule'
Index mode for Indexer Catalog Search was changed from 'Update on Save' to 'Update by Schedule'
Index mode for Indexer Product/Target Rule was changed from 'Update on Save' to 'Update by Schedule'
Index mode for Indexer Target Rule/Product was changed from 'Update on Save' to 'Update by Schedule'
Index mode for Indexer Sales Rule was changed from 'Update on Save' to 'Update by Schedule'
However, the customer grid indexer, it turns out, doesn’t really support update on schedule.
Visualizing sar data with kSar
Published: September 13, 2018
sar is a useful tool that is included in the sysstat package, a set of performance monitoring utilities that is pre-installed on many Linux distros.
Running the sar command prints the gathered data to the terminal in columns.
$ sar 2 10
Linux 4.14.62-v7+ (pluto) 09/13/2018 _armv7l_ (4 CPU)
08:48:05 PM CPU %user %nice %system %iowait %steal %idle
08:48:07 PM all 0.13 0.00 0.00 0.00 0.00 99.87
08:48:09 PM all 0.00 0.00 0.13 0.13 0.00 99.75
08:48:11 PM all 0.38 0.00 0.00 0.00 0.00 99.62
08:48:13 PM all 0.13 0.00 0.13 0.00 0.00 99.75
08:48:15 PM all 0.13 0.00 0.13 0.00 0.00 99.75
08:48:17 PM all 0.00 0.00 0.25 0.00 0.00 99.75
08:48:19 PM all 0.00 0.00 0.38 0.00 0.00 99.62
08:48:21 PM all 0.00 0.00 0.13 0.00 0.00 99.87
08:48:23 PM all 0.00 0.00 0.38 0.00 0.00 99.62
08:48:25 PM all 0.00 0.00 0.00 0.00 0.00 100.00
Average: all 0.08 0.00 0.15 0.01 0.00 99.76
This is useful in some scenarios such as checking CPU usage at a specific period of time. However, in other use cases such as reviewing trends over a longer period of time, this format is not particularly user friendly.
I spent some time recently reviewing the available options for visualizing sar data. While kSar doesn’t offer the most beautiful interface1, it’s easy to install and seems to be the best option available
Here I’ll cover how kSar can be used to visualize sar data.
Reindexing Customer Grid Runs out of Memory in Magento 2
Published: September 10, 2018
The 2.1.X release line of Magento current contains an issue where, with enough customers, reindexing the customer grid will fail with an out of memory fatal error
$ bin/magento indexer:reindex customer_grid
PHP Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 280224918230723 bytes) in /var/www/html/vendor/magento/zendframework1/library/Zend/Db/Statement/Pdo.php on line 228
The issue is fixed in the in the 2.2.X release line and can be fixed in 2.1.X by applying PATCH_MDVA-4538.
Replication Lag Race Conditions in Magento
Published: September 7, 2018
One of my favorite class of bugs are race conditions caused by replication lag (yes, I’m a masochist). These typically play out something like this…
- Application writes data to the master
- Application immediately tries to read that same data back from the slave
- Data hasn’t replicated to slave yet, causing incorrect data to be read
I’ve run into a number of these both in core Magento as well as in 3rd party extensions. In this post I’ll cover a couple of my favorites…
Limiting Access To Specific Tables in MySQL - Cheatsheat
Published: September 6, 2018
One of the core principles in the infosec field is the principle of least privilege. The idea is to limit permitted access by systems or processes as much as humanly possible. Applied to MySQL, in some circumstances this could mean only allowing access to specific tables for some user. This is a quick cheatsheet for working with table-level access in MySQL.