grep color Adding ANSI Escape Sequence

Published: September 19, 2018


Today I was using some command line kung-fu to transform a CSV with thousands of URL redirects into a text file containing the same redirects in Apache mod_rewrite format.

The command looked roughly like this 1

$ cat redirects-with-store-ids.csv | grep ',17' | awk -F"," '{ print "redirect 301 /" $1 "" $2 }' > example-com-redirects.conf

After getting the command just right I opened generated the file in Sublime Text and saw some strange looking characters at the end of each line…

redirect 301 /page/beige.html<0x1b>[01;31m<0x1b>[K
redirect 301 /page/black.html<0x1b>[01;31m<0x1b>[K
redirect 301 /page/blue.html<0x1b>[01;31m<0x1b>[K
redirect 301 /page/brown.html<0x1b>[01;31m<0x1b>[K
redirect 301 /page/green.html<0x1b>[01;31m<0x1b>[K

For some reason “<0x1b>[01;31m<0x1b>[K” had been appended at the end of every line.

Even more curious, when when I used the cat or less command to preview the same file in the terminal the strange characters disappeared…

$ cat example-com-redirects.conf
redirect 301 /page/beige.html
redirect 301 /page/black.html
redirect 301 /page/blue.html
redirect 301 /page/brown.html
redirect 301 /page/green.html

And finally, in TextEdit a I saw a similar, but slightly different sequence of strange characters.

redirect 301 /page/beige.html[01;31m[K
redirect 301 /page/black.html[01;31m[K
redirect 301 /page/blue.html[01;31m[K
redirect 301 /page/brown.html[01;31m[K
redirect 301 /page/green.html[01;31m[K

Here “[01;31m[K” had been appended to the URLs - the same string of characters I saw in Sublime Text, except the “<0x1b>”s were missing

Some Googling brought me to the Stack Exchange question How to create a file using a variable as filename? where I found the following answer

Something is adding color to your output. It might be grep(1), it might adb, it might be baked into the /system/build.prop resource that you’re reading.

If you’re lucky, it is being added by grep(1) – because that is supremely easy to disable with --color=no

Turns out this was caused by the following alias in my ~/.zshrc

alias grep="grep --color=always"

The --color="always" flag was causing grep to add an ANSI color escape sequence at the end of each line.

Re-running the same command, but explicitly passing --color=no to grep fixed the issue for me.


1. Yes, I’m aware that awk -F"," is not exactly a safe thing to do with a CSV (as the data could have commas) but I validated that the data that I was working with didn’t have commas before moving forward with this approach.

Max Chadwick Hi, I'm Max!

I'm a software developer who mainly works in PHP, but loves dabbling in other languages like Go and Ruby. Technical topics that interest me are monitoring, security and performance. I'm also a stickler for good documentation and clear technical writing.

During the day I lead a team of developers and solve challenging technical problems at Something Digital where I mainly work with the Magento platform. I've also spoken at a number of events.

In my spare time I blog about tech, work on open source and participate in bug bounty programs.

If you'd like to get in contact, you can find me on Twitter and LinkedIn.