Redis for Magento Developers

Who Am I?

  • Technical Lead for Strategic Engagement Group at Something Digital (Magento Enterprise Solution Partner)
  • Actively blogging about Magento and other technology
  • Top 50 Magento Contributor 2017

What Will We Cover?

  • The basics: What is Redis? Why Redis? Basic Usage? How Magento uses Redis?
  • The basics: What is Redis? Why Redis? Basic Usage?
  • Intermediate tips: Redis configuration tips. Security concerns
  • Expert level: Performance considerations. Debugging tips
  • Expert level: Performance considerations.

The basics

What is Redis?

Redis is an open source, in-memory data structure store, used as a database, cache and message broker

https://redis.io/

Screenshot of Redis GitHub repo

https://github.com/antirez/redis

Screenshot of antirez.com homepage

http://antirez.com/

Screenshot of Antirez twitter

https://twitter.com/antirez

Why Redis?

  • In memory == FAST
  • Advantages compared memcached (on-disk save, replication, better eviction control...)
  • Most popular cache backend for Magento
    • ## Basic Usage ```bash # Connect $ redis-cli -h HOST -p PORT # Human readable information and statistics 127.0.0.1:6379> INFO redis_version:3.2.8 redis_git_sha1:00000000 redis_git_dirty:0 ... # Flushing a database 127.0.0.1:6379> SELECT 0 OK 127.0.0.1:6379> FLUSHDB OK ```
    ## Be CAREFUL! ```bash # Show all keys - May cause service interruption 127.0.0.1:6379> KEYS * 1) "zc:ti:a04_CHECKOUT_ONEPAGE_PROGRESS_SHIPPING_METHOD" 2) "zc:ti:a04_ADMINHTML_DASHBOARD_INDEX" 3) "zc:k:a04_Zend_LocaleC_en_US_country_CK" ... # Whoops - Cleared all the sessions 127.0.0.1:6379> FLUSHALL OK ```

    How Magento Uses Redis

  • "Object" cache
  • Session storage
  • Page cache (only recommended for Magento 1)

    • Redis Hashes / Hash Commands

  • HGETALL hash
  • HGET hash field
  • HSET hash field "value"
  • HMSET hash field1 "value1" field2 "value2"
    • ## Inspecting the cache ```bash 127.0.0.1:6379> HGETALL zc:k:06f_CORE_CACHE_INVALIDATE 1) "d" 2) "a:1:{s:9:\"full_page\";i:1;}" 3) "t" 4) "06f_MAGE" 5) "m" 6) "1533829835" 7) "i" 8) "1" ```
    ## It may look like this! ```bash 127.0.0.1:6379> HGETALL zc:k:f2e_LAYOUT_10C0D9697E6059BB2948C1CF25BEDE677 1) "i" 2) "1" 3) "t" 4) "f2e_DEFAULT,f2e_CMS_PAGE,f2e_STORE_DEFAULT,f2e_CMS_INDEX_INDEX,f2e_PAGE_TWO_COLUMNS_LEFT,f2e_CUSTOMER_LOGGED_OUT,f2e_LAYOUT_GENERAL_CACHE_TAG,f2e_MAGE" 5) "d" 6) "gz:\x1f\x8bx\x01\xed]\xdd\x8f\xdbHr\x7f\xbe\xfd+\x84y\b\x92\a\x89\xeb=$9 3\x02\xd6\x83\xbd\xac\x0f\xb6\xe3\xac}\x17 /\x04%\xf5H\xf4P\xa4\xcc\x8f\x19\xcf\xfe\xf5\xf9UW\x7f\xb3IQ\x1aM\xceH\x0cc`\xb1\xd9]]]]]_]\xdd\xbc^\x15\xd5\xfa~Vf{qsuW\xd5\xfb{\xf1t5k\x9f\x0ex\\W\xb5HZ\xb1?\x14Y+P\xa8~\xa9\x17\xaa\xf2\xe2\xb0k\xf7\xc5U\xb2\xfc\xe1\xba\xc8V\xa2X\xfe\\\x14\xb3\x0f\xd9V4\xd7\t\x17\xfcp\xcd}0\xd0\x03^%\xb2\x89\xea\xb4\xae\xaa\xf6jVu\xed\xa1ko\xae\xda\xeaW\x02\xe7t&\x1b\xfcq]\x15\xdd\xbelTo\xcb\ ```

    Intermediate tips

    Memory Management

  • Set maxmemory or Redis could consume all the available memory on the box
  • Monitor used memory and configure alerts
  • Use separate redis instances (not separate dbs)
  • Understand maxmemory-policies
    • ## Setting maxmemory ### At runtime ```bash 127.0.0.1:6379> CONFIG SET maxmemory 1073741824 OK ```
    ## Setting maxmemory ### In the config file ```bash $ grep maxmemory /etc/redis/redis.conf maxmemory 1gb ```

    Monitor Memory Usage

    https://newrelic.com/plugins/poison-pen-llc/28

    Monitor Memory Usage

    Configure Alerts

    Use separate instances

  • Tune maxmemory per storage type
  • Reduce FLUSHALL risk
  • Multiple execution threads

    • maxmemory-policies

  • noeviction: NOTHING gets evicted when maxmemory is hit
  • allkeys-lru: Use the LRU algorithm to evict from ALL KEYS (don't need TTL set)
  • volatile-lru: Use the LRU to evict keys with TTL set

    • maxmemory-policies

  • allkeys-random: Randomly evict from ALL KEYS
  • volatile-random: Randomly evict from keys with TTL set
  • volatile-ttl: Evict keys with shortest TTL

    • maxmemory-policies

    allkeys-lru is the safest!

    Security

    http://antirez.com/news/96

    https://maxchadwick.xyz/blog/ssrf-exploits-against-redis

    Security

  • Block public access via firewall
  • Consider adding AUTH (defense in depth, SSRF)

    • Expert Level

    Performance

    "Usually Redis processing time is extremely low, in the sub microsecond range"

    https://redis.io/topics/latency

    Performance

    "It is a good practice to use the ping program to quickly check the latency between the client and server hosts"

    https://redis.io/topics/benchmarks

    Performance

    Magento is very chatty with Redis

    Performance

    Survey of /catalog/category/view for 8 production Magento instances

    ## Network Concerns ```bash $ ping -c 10 172.24.32.122 PING 172.24.32.122 (172.24.32.122) 56(84) bytes of data. 64 bytes from 172.24.32.122: icmp_seq=1 ttl=64 time=0.105 ms 64 bytes from 172.24.32.122: icmp_seq=2 ttl=64 time=0.086 ms 64 bytes from 172.24.32.122: icmp_seq=3 ttl=64 time=0.107 ms 64 bytes from 172.24.32.122: icmp_seq=4 ttl=64 time=0.081 ms 64 bytes from 172.24.32.122: icmp_seq=5 ttl=64 time=0.120 ms 64 bytes from 172.24.32.122: icmp_seq=6 ttl=64 time=0.130 ms 64 bytes from 172.24.32.122: icmp_seq=7 ttl=64 time=0.099 ms 64 bytes from 172.24.32.122: icmp_seq=8 ttl=64 time=0.079 ms 64 bytes from 172.24.32.122: icmp_seq=9 ttl=64 time=0.082 ms 64 bytes from 172.24.32.122: icmp_seq=10 ttl=64 time=0.076 ms --- 172.24.32.122 ping statistics --- 10 packets transmitted, 10 received, 0% packet loss, time 9000ms rtt min/avg/max/mdev = 0.076/0.096/0.130/0.020 ms ```
    ## Network Concerns - `nping` - `redis-cli --latency` - `traceroute` - `tcptraceroute`
    ## Network Concerns Round trip time of more than 0.5ms is a red flag
    ## Performance "Redis uses a mostly single threaded design" https://redis.io/topics/latency#single-threaded-nature-of-redis
    ## Performance "A consequence of being single thread is that when a request is slow to serve all the other clients will wait for this request to be served." https://redis.io/topics/latency#latency-generated-by-slow-commands
    ## Performance "It is possible to monitor slow commands using the [Redis Slow Log](https://redis.io/commands/slowlog) feature." https://redis.io/topics/latency#latency-generated-by-slow-commands
    ## SLOWLOG threshold ### At runtime ```bash 127.0.0.1:6379> CONFIG SET slowlog-log-slower-than 10000 OK ```
    ## SLOWLOG threshold ### In the config file ```bash $ grep slowlog-log-slower-than /etc/redis/redis.conf slowlog-log-slower-than 10000 ```
    ## Reading the slowlog ```bash 127.0.0.1:6379> SLOWLOG GET 9) 1) (integer) 105 2) (integer) 1538060008 3) (integer) 593744 4) 1) "SUNION" 2) "zc:ti:a95_MAGE" ```

    Reading the slowlog

    Every entry is composed of four (or six starting with Redis 4.0) fields:

    • A unique progressive identifier for every slow log entry.
    • The unix timestamp at which the logged command was processed.
    • The amount of time needed for its execution, in microseconds
    • The array composing the arguments of the command.

    https://redis.io/commands/slowlog

    ## Proceed with caution - KEYS * (Use SCAN / write a PHP script) - FLUSHDB (FLUSHDB ASYNC was added in 4.0.0)

    Key Takeaways

    • Understand impact of in-memory storage
    • Secure your redis instance
    • Understand significance of low network RTT
    • Understand single-threaded nature

    Questions?

    Thank You!