grep color Adding ANSI Escape Sequence

Published: September 19, 2018

Tags:

Today I was using some command line kung-fu to transform a CSV with thousands of URL redirects into a text file containing the same redirects in Apache mod_rewrite format.

The command looked roughly like this 1

$ cat redirects-with-store-ids.csv | grep ',17' | awk -F"," '{ print "redirect 301 /" $1 " https://www.example.com/" $2 }' > example-com-redirects.conf

After getting the command just right I opened generated the file in Sublime Text and saw some strange looking characters at the end of each line…

redirect 301 /page/beige.html https://www.example.com/page.html?color=51<0x1b><0x1b>[K
redirect 301 /page/black.html https://www.example.com/page.html?color=52<0x1b>[01;31m<0x1b>
redirect 301 /page/blue.html https://www.example.com/page.html?color=53<0x1b>[01;31m<0x1b>[K
redirect 301 /page/brown.html https://www.example.com/page.html?color=55<0x1b><0x1b>
redirect 301 /page/green.html https://www.example.com/page.html?color=57<0x1b><0x1b>

For some reason “<0x1b>[01;31m<0x1b>[K” had been appended at the end of every line.

Even more curious, when when I used the cat or less command to preview the same file in the terminal the strange characters disappeared…

$ cat example-com-redirects.conf
redirect 301 /page/beige.html https://www.example.com/page.html?color=51
redirect 301 /page/black.html https://www.example.com/page.html?color=52
redirect 301 /page/blue.html https://www.example.com/page.html?color=53
redirect 301 /page/brown.html https://www.example.com/page.html?color=55
redirect 301 /page/green.html https://www.example.com/page.html?color=57

And finally, in TextEdit a I saw a similar, but slightly different sequence of strange characters.

redirect 301 /page/beige.html https://www.example.com/page.html?color=51
redirect 301 /page/black.html https://www.example.com/page.html?color=52
redirect 301 /page/blue.html https://www.example.com/page.html?color=53
redirect 301 /page/brown.html https://www.example.com/page.html?color=55
redirect 301 /page/green.html https://www.example.com/page.html?color=57

Here “[01;31m” had been appended to the URLs - the same string of characters I saw in Sublime Text, except the “<0x1b>”s were missing

Some Googling brought me to the Stack Exchange question How to create a file using a variable as filename? where I found the following answer

Something is adding color to your output. It might be grep(1), it might adb, it might be baked into the /system/build.prop resource that you’re reading.

If you’re lucky, it is being added by grep(1) – because that is supremely easy to disable with --color=no

https://stackoverflow.com/a/8713080/2877224

Turns out this was caused by the following alias in my ~/.zshrc

alias grep="grep --color=always"

The --color="always" flag was causing grep to add an ANSI color escape sequence at the end of each line.

Re-running the same command, but explicitly passing --color=no to grep fixed the issue for me.

Footnotes

1. Yes, I’m aware that awk -F"," is not exactly a safe thing to do with a CSV (as the data could have commas) but I validated that the data that I was working with didn’t have commas before moving forward with this approach.

Max Chadwick Hi, I'm Max!

I'm a software developer who mainly works in PHP, but also dabbles in Ruby and Go. Technical topics that interest me are monitoring, security and performance.

During the day I solve challenging technical problems at Something Digital where I mainly work with the Magento platform. I also blog about tech, work on open source and hunt for bugs.

If you'd like to get in touch with me the best way is on Twitter.