Varnish Developer Portal

Configuring Varnish for Magento

Tags: magento (1) vcl (29)

Magento is one of the most popular e-commerce platforms out there and has both a Community Edition and an Enterprise Edition.

Magento is very flexible and versatile and that comes at a cost: the performance of a Magento store with lots of products and lots of visitors is often quite poor without a decent caching layer.

Luckily Magento has built-in caching mechanisms and the full page cache system has native support for Varnish.

This tutorial is a step-by-step guide on how to configure Varnish for Magento.

1. Install and configure Varnish

If you already have your Magento store up-and-running, and you’re looking to use Varnish as a full-page cache, you’ll have to decide where to install Varnish:

  • You can install Varnish on a dedicated machine and point your DNS records to that server.
  • You can install Varnish on the same server as your Magento store.

For a detailed step-by-step Varnish installation guide, we’d like to refer you to one of the following dedicated tutorials:

2. Reconfigure the web server port

The web server that is hosting your Magento store is most likely set up to handle incoming HTTP requests on port 80. In order for Varnish caching to properly work, Varnish needs to be listening on port 80. This also means that your web server needs to be configured on another port. We’ll use port 8080 as the new web server listening port.

Depending on the type of web server you’re using, different configuration files need to be modified. Here’s a quick how-to for Apache and Nginx.

Apache

If you’re using Apache as your web server you need to replace Listen 80 with Listen 8080 in Apache’s main configuration file.

The individual virtual hosts will also contain port information. You will need to replace <VirtualHost *:80> with <VirtualHost *:8080> in all virtual host files.

Here’s how to change Apache’s listening port for various Linux distributions:

These changes will only take effect once Apache is restarted.

Nginx

If you’re using Nginx, you’ll only have to replace listen 80; with listen 8080; in all virtual host files.

Here’s how to change Nginx’s listening port for various Linux distributions:

These changes will only take effect once Nginx is restarted.

3. Enable Varnish Full-Page Cache in the Magento admin panel

To ensure that Magento is aware of Varnish, you need to enable Varnish as a caching application in Magento’s Full Page Cache configuration.

Here’s how to configure this:

  1. Go to the Magento admin panel
  2. Select Stores > Configuration > Advanced > System
  3. Expand the Full Page Cache section
  4. Select Varnish Cache as the Caching Application
  5. Optionally modify the TTL for public content value to override the standard Time-To-Live
  6. Expand the Varnish Configuration section
  7. Optionally modify the Access list value to override the list of allowed hosts to purge the cache
  8. Optionally modify the Backend host value to override the hostname of your Varnish backend
  9. Optionally modify the Backend port value to override the port number of your Varnish backend
  10. Optionally modify the Grace period value to override the amount of grace time Varnish applies
  11. To confirm the changes, press the Save Config button
  12. Click Export VCL for Varnish 6 to download to custom-generated VCL file.

In this tutorial we’re assuming that Varnish and Magento are hosted on the same server. This means the following default values can be used:

  • The Access list value can be set to localhost
  • The Backend host value can be set to localhost
  • The Backend port value can be set to 8080

Other values, such as TTL for public content and Grace period, can use their respective default values: 86400 for the TTL and 300 for the Grace period.

4. Deploy the custom Magento VCL

In the previous step we downloaded a custom-generated VCL. Please ensure that this file is deployed to /etc/varnish/default.vcl on your Varnish server.

An alternative solution is to run the following command:

sudo bin/magento varnish:vcl:generate --export-version=6 | sudo tee /etc/varnish/default.vcl > /dev/null

This command will regenerate the VCL file using Magento’s command line interface and export it using the Varnish 6 syntax. The output will be written to /etc/varnish/default.vcl, which is the standard VCL file.

Fixing the backend health checks for Magento 2.4

If you’re using Magento 2.4, your web server’s root folder will be configured to the location of your Magento pub folder.

When your Magento root folder would be /var/www/html, the web server’s root will be /var/www/html/pub. In older versions of Magento that was not the case and /var/www/html would have been the root folder.

The problem is that Magento’s generated VCL file still points to the /pub/health_check.php endpoint, as you can see in the VCL snippet below:

backend default {
    .host = "localhost";
    .port = "8080";
    .first_byte_timeout = 600s;
    .probe = {
        .url = "/pub/health_check.php";
        .timeout = 2s;
        .interval = 5s;
        .window = 10;
        .threshold = 5;
   }
}

The probe that performs the health checks will return HTTP 404 errors and as a consequence Varnish will return HTTP 503 Backend fetch failed errors because it considers the backend to be unhealthy.

To fix this, change the .url property for the probe from /pub/health_check.php to /health_check.php as illustrated below:

backend default {
    .host = "localhost";
    .port = "8080";
    .first_byte_timeout = 600s;
    .probe = {
        .url = "/health_check.php";
        .timeout = 2s;
        .interval = 5s;
        .window = 10;
        .threshold = 5;
   }
}

You can also run the following command to perform a find and replace for you:

sudo sed -i "s/\/pub\/health_check.php/\/health_check.php/g" /etc/varnish/default.vcl

Optimized Magento VCL file

Although the generated VCL file is pretty decent once the health check is fixed, there are still some optimizations that can be made.

Here’s the optimized VCL file we recommend:

vcl 4.1;

import std;

backend default {
    .host = "localhost";
    .port = "8080";
    .first_byte_timeout = 600s;
    .probe = {
        .url = "/health_check.php";
        .timeout = 2s;
        .interval = 5s;
        .window = 10;
        .threshold = 5;
   }
}

# Add hostnames, IP addresses and subnets that are allowed to purge content
acl purge {
    "localhost";
    "127.0.0.1";
    "::1";
}

sub vcl_recv {
    # Remove empty query string parameters
    # e.g.: www.example.com/index.html?    
    if (req.url ~ "\?$") {
        set req.url = regsub(req.url, "\?$", "");
    }

    # Remove port number from host header
    set req.http.Host = regsub(req.http.Host, ":[0-9]+", "");
    
    # Sorts query string parameters alphabetically for cache normalization purposes    
    set req.url = std.querysort(req.url);
    
    # Remove the proxy header to mitigate the httpoxy vulnerability
    # See https://httpoxy.org/    
    unset req.http.proxy;

    # Add X-Forwarded-Proto header when using https
    if (!req.http.X-Forwarded-Proto) {
        if(std.port(server.ip) == 443 || std.port(server.ip) == 8443) {
            set req.http.X-Forwarded-Proto = "https";
        } else {
            set req.http.X-Forwarded-Proto = "http";
        }
    }
    
    # Reduce grace to 300s if the backend is healthy
    # In case of an unhealthy backend, the original grace is used
    if (std.healthy(req.backend_hint)) {
        set req.grace = 300s;
    }
    
    # Purge logic to remove objects from the cache
    # Tailored to Magento's cache invalidation mechanism
    if (req.method == "PURGE") {
        if (client.ip !~ purge) {
            return (synth(405, "Method not allowed"));
        }
        if (!req.http.X-Magento-Tags-Pattern && !req.http.X-Pool) {
            return (purge);
        }
        if (req.http.X-Magento-Tags-Pattern) {
          ban("obj.http.X-Magento-Tags ~ " + req.http.X-Magento-Tags-Pattern);
        }
        if (req.http.X-Pool) {
          ban("obj.http.X-Pool ~ " + req.http.X-Pool);
        }
        return (synth(200, "Purged"));
    }
    
    # Only handle relevant HTTP request methods
    if (req.method != "GET" &&
        req.method != "HEAD" &&
        req.method != "PUT" &&
        req.method != "POST" &&
        req.method != "PATCH" &&
        req.method != "TRACE" &&
        req.method != "OPTIONS" &&
        req.method != "DELETE") {
          return (pipe);
    }
    
    # Only cache GET and HEAD requests
    if (req.method != "GET" && req.method != "HEAD") {
        return (pass);
    }

    # Don't cache the health check page
    if (req.url ~ "^/(pub/)?(health_check.php)$") {
        return (pass);
    }

    # Collapse multiple cookie headers into one
    std.collect(req.http.Cookie);

    # Remove tracking query string parameters used by analytics tools
    if (req.url ~ "(\?|&)(_branch_match_id|_bta_[a-z]+|campid|customid|_ga|gclid|gclsrc|gdf[a-z]+|cx|dm_i|ef_id|epik|ie|igshid|cof|hsa_[a-z]+|_ke|mk[a-z]{3}|msclkid|(mtm|matomo)_[a-z]+|pcrid|p(iwi)?k_[a-z]+|redirect(_log)?_mongo_id|siteurl|s_kwcid|sb_referer_host|si|trk_[a-z]+|zanpid|origin|fbclid|mc_[a-z]+|utm_[a-z]+|_bta_[a-z]+)=") {
        set req.url = regsuball(req.url, "(_branch_match_id|_bta_[a-z]+|campid|customid|_ga|gclid|gclsrc|cx|dm_i|ef_id|epik|ie|igshid|cof|hsa_[a-z]+|_ke|mk[a-z]{3}|msclkid|(mtm|matomo)_[a-z]+|pcrid|p(iwi)?k_[a-z]+|redirect(_log)?_mongo_id|siteurl|s_kwcid|sb_referer_host|si|trk_[a-z]+|zanpid|origin|fbclid|mc_[a-z]+|utm_[a-z]+|_bta_[a-z]+)=[-_A-z0-9+()%.]+&?", "");
        set req.url = regsub(req.url, "[?|&]+$", "");
    }

    # Don't cache the authenticated GraphQL requests
    if (req.url ~ "/graphql" && req.http.Authorization ~ "^Bearer") {
        return (pass);
    }

    return (hash);
}

sub vcl_hash {
    # Add a cache variation based on the X-Magento-Vary cookie, but not for graphql requests
    if (req.url !~ "/graphql" && req.http.cookie ~ "X-Magento-Vary=") {
        hash_data(regsub(req.http.cookie, "^.*?X-Magento-Vary=([^;]+);*.*$", "\1"));
    }
    
    # Create cache variations depending on the request protocol
    hash_data(req.http.X-Forwarded-Proto);

    if (req.url ~ "/graphql") {
        # Create cache variations based on the cache ID that is set by Magento
        if (req.http.X-Magento-Cache-Id) {
            hash_data(req.http.X-Magento-Cache-Id);
        } else {
            # If no X-Magento-Cache-Id header is set, use the store and currency values to vary on
            hash_data(req.http.Store);
            hash_data(req.http.Content-Currency);
        }
    }
}

sub vcl_backend_response {
	# Serve stale content for three days after object expiration
	# Perform asynchronous revalidation while stale content is served
    set beresp.grace = 3d;

    # All text-based content can be parsed as ESI
    if (beresp.http.content-type ~ "text") {
        set beresp.do_esi = true;
    }

    # Allow GZIP compression on all JavaScript files and all text-based content
    if (bereq.url ~ "\.js$" || beresp.http.content-type ~ "text") {
        set beresp.do_gzip = true;
    }
    
    # Add debug headers
    if (beresp.http.X-Magento-Debug) {
        set beresp.http.X-Magento-Cache-Control = beresp.http.Cache-Control;
    }

    # Only cache HTTP 200 and HTTP 404 responses
    if (beresp.status != 200 && beresp.status != 404) {
        set beresp.ttl = 120s;
        set beresp.uncacheable = true;
        return (deliver);
    }
    
    # Don't cache if the request cache ID doesn't match the response cache ID for graphql requests
    if (bereq.url ~ "/graphql" && bereq.http.X-Magento-Cache-Id && bereq.http.X-Magento-Cache-Id != beresp.http.X-Magento-Cache-Id) {
       set beresp.ttl = 120s;
       set beresp.uncacheable = true;
       return (deliver);
    }

    # Remove the Set-Cookie header for cacheable content
    # Only for HTTP GET & HTTP HEAD requests
    if (beresp.ttl > 0s && (bereq.method == "GET" || bereq.method == "HEAD")) {
        unset beresp.http.Set-Cookie;
    }
}

sub vcl_deliver {
    # Add debug headers
    if (resp.http.X-Magento-Debug) {
        if (obj.uncacheable) {
            set resp.http.X-Magento-Cache-Debug = "UNCACHEABLE";
        } else if (obj.hits) {
            set resp.http.X-Magento-Cache-Debug = "HIT";
            set resp.http.Grace = req.http.grace;
        } else {
            set resp.http.X-Magento-Cache-Debug = "MISS";
        }
    } else {
        unset resp.http.Age;
    }

    # Don't let browser cache non-static files
    if (resp.http.Cache-Control !~ "private" && req.url !~ "^/(pub/)?(media|static)/") {
        set resp.http.Pragma = "no-cache";
        set resp.http.Expires = "-1";
        set resp.http.Cache-Control = "no-store, no-cache, must-revalidate, max-age=0";
    }
    
    # Cleanup headers
    unset resp.http.X-Magento-Debug;
    unset resp.http.X-Magento-Tags;
    unset resp.http.X-Powered-By;
    unset resp.http.Server;
    unset resp.http.X-Varnish;
    unset resp.http.Via;
    unset resp.http.Link;
}

5. Restart the services

If you’re using Apache as a web server, you’ll run the following command to restart it:

sudo systemctl restart apache2

If you’re using Nginx instead, please run the following command to restart your web server:

sudo systemctl restart nginx

And finally, you’ll have to run the following command to restart Varnish:

sudo systemctl restart varnish

After the restart, your web server will accept traffic on port 8080, Varnish will handle HTTP traffic on port 80. The restart will also ensure the right VCL file is loaded, which will ensure that requests for your Magento store can be properly cached.

6. Making cache purges work

By default Magento will send HTTP PURGE requests to Varnish using the base url. This is not done through the loopback interface but through one of the main network interfaces. The IP address the purge requests originate from doesn’t match localhost and will cause the purge to fail.

Assuming Varnish is installed on the same machine as Magento, listening on port 80, the following command can be executed to make Magento aware of Varnish:

bin/magento setup:config:set --http-cache-hosts=localhost

This will ensure the loopback interface is used and the incoming requests pass the ACL, which is configured to only allow connections coming from localhost.

7. Flushing the cache

In the final step we will flush all Magento caches to ensure a consistent state:

bin/magento cache:flush