Varnish Configuration Language (VCL)

The Varnish Configuration Language (VCL) is a domain-specific programming language used by Varnish to control request handling, routing, caching, and several other aspects.

At first glance, VCL looks a lot like a normal, top-down programming language, with subroutines, if-statements and function calls. However, it is impossible to execute VCL outside of Varnish. Instead, you write code that is run inside Varnish at specific stages of the request-handling process. This lets you define advanced logic that extends the default behavior of Varnish.

Finite State Machine#

The execution of VCL revolves around a Finite State Machine (FSM) that is used throughout the request handling process. As soon as an HTTP request is received, the FSM is triggered.

Depending on the cacheability of responses, or the fact that a request results in a hit, a miss, or a cache bypass, transitions are made to the next state.

Here’s a diagram of the complete Finite State Machine.

• Green lines mark the cache hit flow
• Blue lines mark the cache miss flow
• Red lines mark the cache bypass flow
• The purple line marks the cache purge flow
• Yellow lines mark the piping flow
• Grey lines mark the synthetic response flow

Cache hit flow#

When a request is received that is cacheable, a transition is made from vcl_recv to vcl_hash. If the hash that is created is found in the cache, we’re dealing with a cache hit. This results in a transition to vcl_hit.

If the object is still fresh, the response is delivered to the client through vcl_deliver.

In this scenario no backend connections are made, because the cached object could be delivered to the requesting client.

Cache miss flow#

When a cacheable request is not found in the cache, the cache miss flow is executed.

Just like for any other request, the vcl_recv subroutine is called and just like for cache hits a transition is made to vcl_hash. When the hash doesn’t yield a result, it means the object is not found in the cache and a backend fetch is needed.

Because of the miss, we obviously end up in vcl_miss.

A transition is made to vcl_backend_fetch. A backend HTTP request is prepared and executed. When the backend successfully responds, we transition to vcl_backend_response where the backend HTTP response is turned into a cache object and is stored in the cache.

As soon as the object is stored in the cache, the next request will result in a cache hit.

Eventually a transition to vcl_deliver is made where the HTTP response is returned to the client.

Cache bypass flow#

Not every request results in a hit or a miss. Remember: a cache miss is a hit that hasn’t yet taken place.

When the request is received and processed in vcl_recv, the built-in VCL performs a number of checks to see whether or not the request can be served from cache.

If the requests is not GET or HEAD, the cache will be bypassed because we’re detaling with an uncacheable request method.

The cache is also bypassed when the request contains an Authorization header or a Cookie header. This is also described in the built-in VCL.

A cache bypass will send cause a transition to vcl_hash, but from there we will transition to vcl_pass.

The rest of the cache bypass flow is similar to the cache miss flow:

• A backend HTTP request is prepared and executed in vcl_backend_fetch
• A backend HTTP response is received from the origin server in vcl_backend_response
• The content is delivered to the client through vcl_deliver

An important difference between vcl_pass and vcl_miss is that for requests that bypass the cache, the backend response is not stored in the cache inside vcl_backend_response.

Whereas the cache bypass flow can be triggered by uncacheable requests, the flow is also used when a backend HTTP response is considered uncacheable. The built-in VCL for vcl_backend_response does a number of checks on the response it gets from the origin server. If it turns out that the response is not cacheable, a Hit-For-Miss is triggered.

A Hit-For-Miss will cache the decision not to serve the object from cache for a period of 2 minutes or until the next response is cacheable. When a cache lookup takes place in vcl_hash, objects that are marked as Hit-For-Miss will cause a transition to vcl_pass.

Cache purge flow#

Although objects will expire from the cache and be updated after revalidation, sometimes this is not an option and content needs to be removed from the cache immediately.

Actively purging objects from the cache is supported by Varnish and can be triggered by calling return( purge ); in vcl_recv.

This will cause a transition to vcl_hash and when the object is found and removed from the cache, the Finite State Machine will transition to vcl_purge where synthetic HTTP/1.1 200 Purged response is returned.

Because the response is a synthetic one, no backend logic is executed and a transition to vcl_synth is made where the return values are parsed into the output template. From there the content is returned to the requesting client.

Piping flow#

The majority of the uncached or uncacheable requests will hit the backend flow through subroutines like vcl_backend_fetch and vcl_backend_response.

But when Varnish isn’t sure whether the incoming requests is actually an HTTP request, it abandons the regular HTTP flow and downgrades to a basic TCP flow. This means that there is no more awareness of HTTP and bytes are simply shuffled over the wire.

A typical use case is the use of invalid requests methods, as desribed in the built-in VCL. When a non-existent HTTP method is used, Varnish no longer believes it is an HTTP request and transitions from vcl_recv to vcl_pipe.

The payload is forwarded to the backend and the connection is closed immediately, preventing connection reuse to take place.

Synthetic response flow#

Synthetic responses are HTTP responses that did not originate from the origin server. Instead the output is generated by Varnish and returned directly to the requesting client.

The built-in VCL for vcl_recv has a couple of scenarios where synthetic output is returned. But in the cache purge flow we already showed that vcl_purge returns a synthetic HTTP/1.1 200 Purged response.

Triggering a synthetic response is done by calling return( synth(200,"Hello world") ). This example will return a synthetic HTTP/1.1 200 Hello world response. This causes a transition to vcl_synth where the status code and reason is parsed into the output template.

From vcl_synth, we immediately return the synthetic response to the requesting client.

VCL subroutines and return actions#

As described in built-in subroutines section of the Varnish Cache User Guide, a number of built-in subroutines are available.

They are executed when their corresponding state is reached in the Finite State Machine. Although each of these subroutines has a built-in VCL behavior, its logic can be altered by writing custom VCL.

Client-side subroutines#

The client-side subroutines are the parts of the execution flow that do not interact with the backend. Their execution path is the fastest, because the don’t have to wait for the potentially slow backends to respond.

The cache hit flow is a perfect example of an execution path that only transitions to client-side subroutines.

vcl_recv#

The vcl_recv subroutine is called when a request is received. This could be a regular HTTP request coming from a client, this could be an internal ESI subrequest, or even a restart of a previous request.

The following return actions are supported in vcl_recv:

• return(fail): triggers a fatal error. The equivalent of return(synth(503, "VCL Failed"))
• return(synth(status code, reason)): returns synthetic output using the returned status code and reason phrase.
• return(restart): restarts the transaction, increases the restart counter and fails if max_restarts is reached
• return(pass): bypass the cache
• return(pipe): switch to pipe mode
• return(hash): lookup the object in the cache
• return(purge): purge the object and its variants from the cache
• return(vcl(label)): load a labeled VCL configuration and call its vcl_recv subroutine

vcl_pipe#

The vcl_pipe subroutine is triggered when pipe mode is enabled by calling return(pipe) inside vcl_recv.

As described in the piping flow, pipe mode will degrade the incoming request to plain TCP and no longer consider it to be an HTTP request. The bytes are shuffled over the wire and the connection is closed.

The following return actions are supported in vcl_pipe:

• return(fail): triggers a fatal error. The equivalent of return(synth(503, "VCL Failed"))
• return(synth(status code, reason)): returns synthetic output using the returned status code and reason phrase.
• return(pipe): proceed with pipe mode and send the data directly to the origin server

vcl_pass#

The vcl_pass subroutine is triggered when the cache is bypassed by calling return(pass).

As described in the cache bypass flow, uncacheable requests will bypass the cache and send the incoming HTTP request to the origin server. The returned response is not stored in the cache.

• return(fail): triggers a fatal error. The equivalent of return(synth(503, "VCL Failed"))
• return(synth(status code, reason)): returns synthetic output using the returned status code and reason phrase.
• return(restart): restarts the transaction, increases the restart counter and fails if max_restarts is reached
• return(fetch): initiate a backend request and transition to vcl_backend_fetch

vcl_hash#

The vcl_hash subroutine is responsible for creating the hash key and looking up the object in the cache. The subroutine is called directly after vcl_recv and the support return actions are:

• return(fail): triggers a fatal error. The equivalent of return(synth(503, "VCL Failed"))
• return(lookup): lookup the object in the cache using the hash key that was created

As described in the cache hit flow, a transition to vcl_hit is made when the object is found.

As described in the cache miss flow, a transition to vcl_miss is made when the object is found.

A transition to vcl_miss also takes place if the object is a Hit-For-Miss object.

As described in the cache bypass flow, a transition to vcl_pass is made when the object bypass the cache.

vcl_purge#

The vcl_purge subroutine is reached when return(purge) is called in vcl_recv to purge an object from the cache. This is also mentioned in the cache purge flow.

vcl_purge supports the following return actions:

• return(fail): triggers a fatal error. The equivalent of return(synth(503, "VCL Failed"))
• return(synth(status code, reason)): returns synthetic output using the returned status code and reason phrase.
• return(restart): restarts the transaction, increases the restart counter and fails if max_restarts is reached

vcl_miss#

The vcl_miss subroutine is called when an object was not found in the cache or if the object was considered stale in vcl_hit.

As described in the cache miss flow, vcl_miss will transition to vcl_backend_fetch and fetch the missing or outdated content from the origin server.

vcl_miss supports the following return actions:

• return(fail): triggers a fatal error. The equivalent of return(synth(503, "VCL Failed"))
• return(synth(status code, reason)): returns synthetic output using the returned status code and reason phrase.
• return(restart): restarts the transaction, increases the restart counter and fails if max_restarts is reached
• return(pass): decide to bypass the cache
• return(fetch): initiate a backend request and transition to vcl_backend_fetch

vcl_hit#

The vcl_hit subroutine is called when the cache lookup is successful.

As described in the cache hit flow, vcl_hit will return the cached object to the client by transitioning to vcl_deliver.

vcl_hit supports the following return actions:

• return(fail): triggers a fatal error. The equivalent of return(synth(503, "VCL Failed"))
• return(synth(status code, reason)): returns synthetic output using the returned status code and reason phrase.
• return(restart): restarts the transaction, increases the restart counter and fails if max_restarts is reached
• return(pass): decide to bypass the cache
• return(miss): perform a synchronous revalidation with the origin server for expired and out of grace objects
• return(deliver): deliver the object to the client. For stale content a asynchronous backend fetch is triggered while stale content is returned.

vcl_deliver#

The vcl_deliver subroutine is called just before a response is returned to the requesting client.

vcl_deliver supports the following return actions:

• return(fail): triggers a fatal error. The equivalent of return(synth(503, "VCL Failed"))
• return(synth(status code, reason)): returns synthetic output using the returned status code and reason phrase.
• return(restart): restarts the transaction, increases the restart counter and fails if max_restarts is reached
• return(deliver): deliver the object to the client

vcl_synth#

The vcl_synth subroutine delivers synthetic content to the client. This is content that did not originate from the origin server, but that was generated in VCL instead.

vcl_synth supports the following return actions:

• return(fail): triggers a fatal error. The equivalent of return(synth(503, "VCL Failed"))
• return(restart): restarts the transaction, increases the restart counter and fails if max_restarts is reached
• return(deliver): deliver the synthetic response to the client

Backend-side subroutines#

The backend-side subroutines are responsible for interacting with the origin server. There are only three of them and in the diagram they are part of the grey box where the cache miss flow and the cache bypass flow lead to.

The backend-side routines are generally slower than their client-side counterparts, but that is mainly caused by the fact that these subroutines have to wait for data to be sent over the wire.

vcl_backend_fetch#

The vcl_backend_fetch subroutine is responsible for sending HTTP requests to the origin server and triggers the backend fetch.

vcl_backend_fetch supports the following return actions:

• return(fail): triggers a fatal error. The equivalent of return(synth(503, "VCL Failed"))
• return(fetch): fetch the object from the backend
• return(error(status code, reason)): returns a synthetic error under the form of a backend error by transitioning to vcl_backend_error
• return(abandon): abandon the backend request and transition to vcl_synth unless the request is background fetch. The equivalent of return(synth(503, "VCL Failed"))

vcl_backend_response#

The vcl_backend_response subroutine is called when Varnish receives the HTTP response as a result from the backend fetch that took place in vcl_backend_fetch.

In vcl_backend_response a number of checks are made that decide whether or not the object is stored in the cache and how long is it cached for.

As described in the cache bypass flow, content that is deemed uncacheable is marked as Hit-For-Miss.

vcl_backend_response supports the following return actions:

• return(fail): triggers a fatal error. The equivalent of return(synth(503, "VCL Failed"))
• return(deliver): fetch the object body from the backend and deliver content to any waiting client request. This can be done in parallel and streaming is supported
• return(retry): retry the backend transaction, increases the retries counter and fails if max_retries is reached
• return(error(status code, reason)): returns a synthetic error under the form of a backend error by transitioning to vcl_backend_error
• return(pass(duration)): enables Hit-For-Pass for uncacheable objects, which causes subsequent requests for this resource to transition from vcl_recv to vcl_pass.
• return(abandon): abandon the backend request and transition to vcl_synth unless the request is background fetch. The equivalent of return(synth(503, "VCL Failed"))

vcl_backend_error#

The vcl_backend_error subroutine is called when a backend fetch failure occurs or when the max_retries counter has been exceeded.

Inside vcl_backend_error, a synthetic response is created using an output template that is identical to the output template that is generated by the vcl_synth subroutine.

The only difference is that vcl_backend_error is a backend-side subroutine, which means other variables are parsed. Whereas vcl_synth parses in variables like resp.status and resp.reason, the vcl_backend_error parses in beresp.status and beresp.reason.

vcl_backend_error supports the following return actions:

• return(fail): triggers a fatal error. The equivalent of return(synth(503, "VCL Failed"))
• return(deliver): deliver the synthetic error message as if it was fetched by the backend
• return(retry): retry the backend transaction, increases the retries counter and fails if max_retries is reached

VCL initialization subroutines#

vcl_init#

The vcl_init subroutine is called when the VCL is loaded, beofre any requests are received. This subroutine is mainly used to initialize Varnish modules.

vcl_init supports the following return actions:

• return(ok): VCL initialization was successful
• return(fail): VCL initialization aborted

vcl_fini#

The vcl_fini subroutine is called when the VCL is discarded after all the incoming requests have exited the VCL. You would use this subroutine to handle Varnish modules that require cleanup.

The only return action that vcl_fini supports is return(ok).

VCL syntax#

VCL code is executed inside the various subroutines that we just covered. But before you can start writing VCL, you need to be familiar with its syntax.

VCL version declaration#

Every VCL file starts with a version declaration. As of Varnish 6, the version declaration you’ll want to use is the following:

vcl 4.1;

The vcl 4.0; declaration will still work on Varnish 6, but it prevents some specific Varnish 6 features from being supported.

Assigning values#

The VCL language doesn’t require you to program the full behavior, but rather allows you to extend pre-existing built-in behavior. Given this scope and purpose, the main objective is to set values based on certain conditions.

Let’s take the following line of code for example:

set resp.http.Content-Type = "text/html; charset=utf-8";

It comes from the vcl_synth built-in VCL and assigns the content type text/html; charset=utf-8 to the HTTP response header resp.http.Content-Type.

We’re basically assigning a string to a variable. The assigning is done by using the set keyword. If we want to unset a variable, we just use the unset keyword.

Let’s illustrate the unset behavior with another example:

unset bereq.body;

We’re unsetting the bereq.body variable. This is part of the vcl_backend_fetch logic of the built-in VCL.

Strings#

VCL supports various data types, but the string type is by far the most common one.

Here’s a conceptual example:

set variable = "value";

This is the easiest way to assign a string value. But as soon as you want to use newlines or double quotes, you’re in trouble.

Luckily there’s an alternative, which is called long strings. A long string begins with {" and ends with "}.

They may include newlines, double quotes, and other control characters, except for the NULL (0x00) character.

A very familiar usage of this is the built-in VCL implementation of vcl_synth, where the HTML template is composed using long strings:

set resp.body = {"<!DOCTYPE html>
<html>
  <head>
    <title>"} + resp.status + " " + resp.reason + {"</title>
  </head>
  <body>
    <h1>Error "} + resp.status + " " + resp.reason + {"</h1>
    <p>"} + resp.reason + {"</p>
    <h3>Guru Meditation:</h3>
    <p>XID: "} + req.xid + {"</p>
    <hr>
    <p>Varnish cache server</p>
  </body>
</html>
"};

There is also an alternate form of a long string, which can be delimited by triple double quotes, """...""".

This example also shows how to perform string concatenation and variable interpolation. Let’s reimagine the vcl_synth example, and create a version using simple strings:

set beresp.body = "Status: " + resp.status + ", reason: " + resp.reason";

And again we’re using the +-sign for string concatenation and variable interpolation.

Conditionals#

Although the VCL language is limited in terms of control structures, it does provide conditionals, meaning if/else statements.

Let’s take some built-in VCL code as an example since we’re so familiar with it:

if (req.method != "GET" && req.method != "HEAD") {
    /* We only deal with GET and HEAD by default */
    return (pass);
}

This is just a regular if-statement. We can also add an else clause:

if (req.method != "GET" && req.method != "HEAD") {
    /* We only deal with GET and HEAD by default */
    return (pass);
} else {
    return (hash);
}

And as you’d expect, there’s also an elseif clause:

if (req.method == "GET") {
    return (hash);
} elseif (req.method == "HEAD") {
    return (hash);
} else {
	return (pass);
}

Operators#

VCL has a number of operators that either evaluate to true or to false:

• The = operator is used to assign values.
• The ==, !=, <, <=, >, and >= operators are used to compare values.
• The ~ operator is used to match values to a regular expression or an ACL.
• The ! operator is used for negation.
• && is the logical and operator.
• || is the logical or operator.

And again the built-in VCL comes to the rescue to clarify how some of these operators can be used:

if (req.method != "GET" && req.method != "HEAD") {
    /* We only deal with GET and HEAD by default */
    return (pass);
}

You can clearly see the negation and the logical and, meaning that the expression only evaluates to true when condition one and condition two are false.

We’ve already used the = operator to assign values, but here’s another example for reference:

set req.http.X-Forwarded-Proto = "https";

This example assigns the https value to the X-Forwarded-Proto request header.

A logical or looks like this:

if(req.method == "POST" || req.method == "PUT") {
    return(pass);
}

At least one of the two conditions has to be true for the expression to be true.

And let’s end this part with a less than or equals example:

if(beresp.ttl <= 0s {
    set beresp.uncacheable = true;
}

Comments#

Documenting your code with comments is usually a good idea, and VCL supports three different comment styles.

We’ve listed all three of them in the example below:

sub vcl_recv {
    // Single line of out-commented VCL.
    # Another way of commenting out a single line.
    /*
        Multi-line block of commented-out VCL.
    */
}

So you can use // or # to create a single-line comment. And /* ... */ can be used for multi-line comments.

Numbers#

VCL supports numeric values, both integers and real numbers.

Certain fields are numeric, so it makes sense to assign literal integer or real values to them.

Here’s an example:

set resp.status = 200;

But most variables are strings, so these numbers get cast into strings. For real numbers, their value is rounded to three decimal places (e.g. 3.142).

Booleans#

Booleans can be either true or false. Here’s an example of a VCL variable that expects a boolean:

set beresp.uncacheable = true;

When evaluating values of non-boolean types, the result can also be a boolean.

For example strings will evaluate to true or false if their existence is checked. This could result in the following example:

if(!req.http.Cookie) {
    //Do something
}

if(req.http.Authorization) {
    //Do something
}

Integers will evaluate to false if their value is 0; the same applies to duration types when their values are zero or less.

Boolean types can also be set based on the result of another boolean expression:

set beresp.uncachable = (beresp.http.do-no-cache == "true");

Time and durations#

Time is an absolute value, whereas a duration is a relative value. However, in Varnish they are often combined.

Time#

You can add a duration to a time, which results in a new time:

set req.http.tomorrow = now + 1d;

The now variable is how you can retrieve the current time in VCL. The now + 1d statement means we’re adding a day to the current time. The returned value is also a time type.

But since we’re assigning a time type to a string field, the time value is cast to a string, which results in the following string value:

Thu, 10 Sep 2020 12:34:54 GMT

Duration#

As mentioned, durations are relative. They express a time change and are expressed numerically, but with a time unit attached.

Here are a couple of examples that illustrate the various time units:

• 1ms equals 1 millisecond.
• 5s equals 5 seconds.
• 10m equals 10 minutes.
• 3h equals 3 hours.
• 9d equals 9 days.
• 4w equals 4 weeks.
• 1y equals 1 year.

In string context their numeric value is kept, and the time unit is stripped off. This is exactly what a real number looks like when cast to a string. And just like real numbers, they are rounded to three decimal places (e.g. 3.142).

Here’s an example of a VCL variable that supports durations:

set beresp.ttl = 1h;

So this example sets the TTL of an object to one hour.

Regular expressions#

Pattern matching is a very common practice in VCL. That’s why VCL supports Perl Compatible Regular Expressions (PCRE), and we can match values to a PCRE regex through the ~ operator:

if(req.url ~ "^/[a-z]{2}/cart") {
	return(pass);
}

This example is matching the request URL to a regex pattern that looks for the shopping cart URL of a website. This URL is prefixed by two letters, which represent the user’s selected language. When the URL is matched, the request bypasses the cache.

Backends#

Varnish is a proxy server and depends on an origin server to provide (most of) the content. A backend definition is indispensable, even if you end up serving synthetic content.

This is what a backend looks like:

backend default {
    .host = "127.0.0.1";
    .port = "8080";
}

It has a name, default in this case, and uses the .host and .port properties to define how Varnish can connect to the origin server. The first backend that is defined will be used.

If you’re not planning to use a backend, or if you are using a module for dynamic backends, you’ll have to define the following backend configuration:

backend default none;

This bypasses the requirement that you must define a single backend in your VCL.

Backends also support the following options:

• .connect_timeout is how long to wait for a connection to be made to the backend.
• .first_byte_timeout is how long to wait for the first byte of the response.
• .between_bytes_timeout is the maximum time to wait between bytes when reading the response.
• .last_byte_timeout is the total time to wait for the complete backend response.
• .max_connections is the maximum number of concurrent connections Varnish will hold to the backend. When this limit is reached, requests will fail into vcl_backend_error.

Probes#

Knowing whether or not a backend is healthy is important. It helps to avoid unnecessary outages and allows you to use a fallback system.

When using probes, you can perform health checks at regular intervals. The probe sets the internal value of the health of that backend to healthy or sick.

Backends that are sick always result in an HTTP 503 error when called.

If you use vmod_directors to load balance with multiple backends, sick backends will be removed from the rotation until their health checks are successful and their state changes to healthy.

A sick backend will become healthy when a threshold of successful polls is reached within a polling window.

This is how you define a probe:

probe healthcheck {
}

Default values#

The probe data structure has a bunch of attributes; even without mentioning these attributes, they will have a default behavior:

• .url is the URL that will be polled. The default value is /.
• .expected_response is the HTTP status code to that the probe expects. The default value is 200.
• .timeout is the amount of time the probe is willing to wait for a response before timing out. The default value is 2s.
• .interval is the polling interval. The default value is 5s.
• .window is the number of polls that are examined to determine the backend health. The default value is 8.
• .initial is the number of polls in .window that have to be successful before Varnish starts. The default value is 2.
• .threshold is the number of polls in .window that have to be successful to consider the backend healthy. The default value is 3.
• .tcponly is the mode of the probe. When enabled with 1, the probe will only check for available TCP connections. The default value is 0. This property is only available in Varnish Enterprise.

Extending values#

You can start extending the probe by assigning values to these defaults.

Here’s an example:

probe healthcheck {
    .url = "/health";
    .interval = 10s;
    .timeout = 5s;
}

This example will call the /health endpoint for polling and will send a health check every ten seconds. The probe will wait for five seconds before it times out.

Customizing the entire HTTP request#

When the various probe options do not give you enough flexibility, you can even choose to fully customize the HTTP request that the probe will send out.

The .request property allows you to do this. However, this property is mutually exclusive with the .url property.

Here’s an example:

probe healtcheck {
    .request =
        "HEAD /health HTTP/1.1"
        "Host: localhost"
        "Connection: close"
        "User-Agent: Varnish Health Probe";
    .interval = 10s;
    .timeout = 5s;
}

Although a lot of values remain the same, there are two customizations that are part of the request override:

• The request method is HEAD instead of GET.
• We’re using the custom Varnish Health Probe User-Agent.

Assigning the probe to a backend#

Once your probe is set up and configured, you need to assign it to a backend.

It’s a matter of setting the .probe property in your backend to the name of the probe, as you can see in the example below:

vcl 4.1;

probe healthcheck {
    .url = "/health";
    .interval = 10s;
    .timeout = 5s;
}

backend default {
    .host = "127.0.0.1";
    .port = "8080";
    .probe = healthcheck;
}

By defining your probe as a separate data structure, it can be reused when multiple backends are in use.

The verbose approach is to define the .probe property inline, as illustrated in the example below:

vcl 4.1;

backend default {
    .host = "127.0.0.1";
    .port = "8080";
    .probe = {
	    .url = "/health";
   	  .interval = 10s;
	    .timeout = 5s;
    }
}

TCP-only probes#

Probes usually perform HTTP requests to check the health of a backend. By using TCP-only probes, the health of a backend is checked by the availability of the TCP connection.

This can be used to probe non-HTTP endpoints. However, TCP-only probes cannot be used with .url, .request, or .expected_response properties.

Here’s how you define such a probe:

probe tcp_healtcheck {
  .tcponly = 1;
}

UNIX domain sockets#

The backend data structure has additional properties that can be set with regard to the endpoint it is connecting to.

If you want to connect to your backend using a UNIX domain socket, you’ll use the .path property. It is mutually exclusive with the .host property and is only available when you use the vcl 4.1; version declaration.

Here’s an example of a UDS-based backend definition:

backend default {
    .path = "/var/run/some-backend.sock";
}

Overriding the host header#

If for some reason the Host header is not set in your HTTP requests, you can use the .host_header property to override it.

Here’s an example:

backend default {
    .host = "127.0.0.1";
    .port = "8080";
    .host_header = "example.com";
}

This .host_header property will be used for both regular backend requests and health probe checks.

Access control lists#

An access control list (ACL) is a VCL data structure that contains hostnames, IP addresses, and subnets. An ACL is used to match client addresses and restrict access to certain resources.

Here’s how you define an ACL:

acl admin {
	"localhost";
	"secure.my-server.com";
	"192.168.0.0/24";
	! "192.168.0.25";	
}

This ACL named admin contains the following rules:

• Access from localhost is allowed.
• Access from the hostname secure.my-server.com is also allowed.
• All IP address in the 192.168.0.0/24 subnet are allowed.
• The only IP address from that range that is not allowed is 192.168.0.25.

In your VCL code, you can then match the client IP address to that list, as you’ll see in the next example:

acl admin {
	"localhost";
	"secure.my-server.com";
	"192.168.0.0/24";
	! "192.168.0.25";	
}

sub vcl_recv {
	if(req.url ~ "^/admin/?" && client.ip !~ admin) {
	    return(synth(403,"Forbidden"));
	}
}

In this example, we’re hooking into vcl_recv to intercept requests for /admin or any subordinate resource of /admin/. If users try to access this resource, we check if their client IP address is matched by acl admin.

If it doesn’t match, an HTTP 403 Forbidden error is returned synthetically.

Functions#

Complex logic in a programming language is usually abstracted away by functions. This is also the case in VCL, which has a number of native functions.

The number of functions is limited, but extra functions are available in the wide range of VMODs that are supported by Varnish.

ban()#

ban() is a function that adds an expression to the ban list. These expressions are matched to cached objects. Every matching object is then removed from the cache.

In essence, the ban() function exists to invalidate multiple objects at the same time.

Here’s a quick example:

ban("obj.age > 1h");

Multiple expressions can be chained using the && operator.

hash_data()#

The hash_data() function is used within the vcl_hash subroutine and is used to append string data to the hash input that is used to lookup an object in cache.

Here’s some code from the built-in VCL for vcl_hash where hash_data() is used:

sub vcl_hash {
    hash_data(req.url);
    if (req.http.host) {
        hash_data(req.http.host);
    } else {
        hash_data(server.ip);
    }
    return (lookup);
}

synthetic()#

The synthetic() function prepares a synthetic response body and uses a string argument for its input. This function can be used within vcl_synth and vcl_backend_error.

Here’s an example for vcl_synth:

synthetic(resp.reason);

regsub()#

The regsub() function is a very popular function in Varnish. This function performs string substitution using regular expressions. Basically, do find/replace on the first occurrence using a regex pattern.

This is the API of this function:

regsub(string, regex, sub)

• The string argument is your input.
• The regex argument is the regular expression you’re using to match what you’re looking for in the input string.
• The sub argument is what the input string will be substituted with.

Here’s a practical example where we use regsub() to extract a cookie value:

vcl 4.1;

sub vcl_hash {
    hash_data(regsub(req.http.Cookie,"(;|^)language=([a-z]{2})(;|$)","\2"));
}

This vcl_hash subroutine is used to extend the built-in VCL and to add the value of the language cookie to the hash. This creates a cache variation per language.

We really don’t want to hash the entire cookie because that will drive our hit rate down, especially when there are tracking cookies in place.

In order to extract the exact cookie value we need, we’ll match the req.http.Cookie header to a regular expression that uses grouping. In the substitution part, we can refer to those groups to extract the value we want.

Here’s the regular expression:

(;|^)language=([a-z]{2})(;|$)

This regular expression looks for a language= occurrence, followed by two letters. These letters represent the language. This language cookie can occur at the beginning of the cookie string, in the middle, or at the end. The (;|^) and (;|$) statements ensure that this is possible.

Because we’re using parentheses for grouping, the group where we match the language itself, is indexed as group two. This means we can refer to it in the regsub() function as \2.

So if we look at the entire regsub() example:

regsub(req.http.Cookie,"(;|^)language=([a-z]{2})(;|$)","\2");

And let’s imagine this is our Cookie header:

Cookie: privacy_accepted=1;language=en;sessionid=03F1C5944FF4

Given the regular expression and the group referencing, the output of this regsub() function would be en.

This means that en will be added to the hash along with the URL and the host header.

When the Cookie header doesn’t contain a language cookie, an empty string is returned. When there is no Cookie header, an empty string is returned as well. This means we don’t risk hash-key collisions when the cookie isn’t set.

regsuball()#

The regsuball() function is very similar to the regsub() function we just covered. The only difference is where regsub() matches and replaces the first occurrence of the pattern, regsuball() matches all occurrences.

Even the function API is identical:

regsuball(string, regex, sub)

• The string argument is your input.
• The regex argument is the regular expression you’re using to match what you’re looking for in the input string.
• The sub argument is what the input string will be substituted with.

Here’s a similar example, where we’ll strip off some cookies again. Instead of matching the values we want to keep, we’ll match the values we want to remove. We need to ensure that all occurrences are matched, not just the first occurrence. That’s why we use regsuball() instead of regsub():

regsuball(req.http.Cookie,"_g[a-z0-9_]+=[^;]*($|;\s*)","");

What this example does, is remove all Google Analytics cookies. This is the list of cookies we need to remove:

• _ga
• _gid
• _gat
• _gac_<property-id>

Instead of stripping them off one by one, we can use the _g[a-z0-9_]+=[^;]*($|;\s*) regular expression to match them all at once. In the end we’ll replace the matched cookies with an empty string.

This could be the raw value of your req.http.Cookie header:

cookie1=a; _ga=GA1.2.1915485056.1587105100;cookie2=b; _gid=GA1.2.873028102.1599741176; _gat=1

And the end result is the following:

cookie1=a;cookie2=b

Subroutines#

Besides the built-in VCL subroutines you can also define your own subroutines. Here’s an example:

vcl 4.1;

sub skipadmin {
    if(req.url ~ "^/admin/?") {
        return(pass);
    }
}

sub vcl_recv {
    call skipadmin;
}

The skipadmin subroutine is entirely custom and is called within vcl_recv using the call statement. The purpose of custom subroutines is to allow code to be properly structured and functionality compartmentalized.

Include#

Not all of your VCL logic should necessarily be in the same VCL file. When the line count of your VCL file increases, readability can become an issue.

To tackle this issue, VCL allows you to include VCL from other files. The include syntax is not restricted to subroutines and fixed language structures, even individual lines of VCL code can be included.

The include "<filename>;" syntax will tell the compiler to read the file and copy its contents into the main VCL file.

When including a file, the order of execution in the main VCL file will be determined by the order of inclusion.

This means that each include can define its own VCL routing logic and if an included file exits the subroutine early, it will bypass any logic that followed that return statement.

Let’s apply this to our previous example, where the skipadmin subroutine is used and put the custom subroutine in a separate file:

#This is skipadmin.vcl

sub skipadmin {
    if(req.url ~ "^/admin/?") {
        return(pass);
    }
}

In your main VCL file, you’ll use the include syntax to include skipadmin.vcl:

vcl 4.1;

include "skipadmin.vcl";

sub vcl_recv {
    call skipadmin;
}

Import#

The import statement can be used to import VMODs. These are Varnish modules, written in C-code, that are loaded into Varnish and offer a VCL interface. These modules basically enrich the VCL syntax without being part of the Varnish core.

Here’s a quick example:

vcl 4.1;

import std;

sub vcl_recv {
    set req.url = std.querysort(req.url);
}

This example uses import std; to import Varnish’s standard library containing a set of utility functions. The std.querysort() function will alphabetically sort the query string parameters of a URL, which has a beneficial impact on the hit rate of the cache.

VCL objects and variables#

VCL provides a set of variables that can be set, unset, retrieved or modified. These variables are part of the various VCL objects.

You can group the variables as follows:

• Connection-related variables. Part of the local, server, remote and client objects
• Request-related variables. Part of the req and req_top objects
• Backend request-related variables. Part of the bereq object
• Backend response-related variables. Part of the beresp object
• Cache object-related variables. Part of the obj object
• Response-related variables. Part of the resp object
• Session-related variables. Part of the sess object
• Storage-related variables. Part of the storage object

The full list of VCL objects and variables can be found in VCL variable documentation on varnish-cache.org.

Connection variables#

There are four connection-related objects available in VCL, and their meaning depends on your topology:

• client: the client that sends the HTTP request
• server: the server that receives the HTTP request
• remote: the remote end of the TCP connection in Varnish
• local: the local end of the TCP connection in Varnish

Local variables#

The local object has two interesting variables:

• local.endpoint
• local.socket

Both of these variables are only available using the vcl 4.1 syntax.

local.endpoint contains the socket address for the -a socket.

If -a http=:80 was set in varnishd, the local.endpoint value would be :80.

Whereas local.endpoint takes the socket value, local.socket will take the socket name. If we take the example where -a http=:80 is set, the value for local.socket will be http.

Identities#

Both the client and the server object have an identity variable that identifies them.

client.identity identifies the client, and its default value is the same value as client.ip. This is a string value, so you can assign what you like.

Naturally, the server.identity variable will identify your server. If you specified -i as a runtime parameter in varnishd, this will be the value. If you run multiple Varnish instances on a single machine, this is quite convenient.

But by default, server.identity contains the hostname of your server, which is the same value as server.hostname.

Request variables#

The req object allows you to inspect and manipulate incoming HTTP requests.

The most common variable is req.url, which contains the request URL. But there’s also req.method to get the request method. And every request header is accessible through req.http.*.

Imagine the following HTTP request:

GET / HTTP/1.1
Host: localhost
User-Agent: curl
Accept: */*

Varnish will populate the following request variables:

• req.method will be GET.
• req.url will be /.
• req.proto will be HTTP/1.1.
• req.http.host will be localhost.
• req.http.user-agent will be curl.
• req.http.accept will be */*.
• req.can_gzip will be false because the request didn’t contain an Accept-Encoding: gzip header.

Top-level requests and Edge Side Includes#

Edge Side Includes (ESI) are markup tags that are used to assemble content on the edge, using a source attribute that refers to an HTTP endpoint.

Here’s an ESI example:

<esi:include src="/header" />

ESI tags are parsed by Varnish and the HTTP endpoint in the source attribute is called by an internal subrequest.

At the request level, there is a req.esi_level variable to check the level of depth and a req.esi variable to toggle ESI support.

When you’re in an ESI subrequest, there is also some context available about the top-level request that initiated the subrequest. The req_top object provides that context.

• req_top.url returns the URL of the parent request.
• req_top.http.* contains the request headers of the parent request.
• req_top.method returns the request method of the parent request.
• req_top.proto returns the protocol of the parent request.

If you want to know whether or not the top-level request was requesting the homepage, you could use the following VCL code:

vcl 4.1;
import std;

sub vcl_recv {
	if(req.esi_level > 0 && req_top.url == "/") {
		std.log("ESI call for the homepage");
	}
}

This example will log ESI call for the homepage to Varnish’s Shared Memory Log using the std.log() function.

Backend request variables#

Backend requests are the requests that Varnish sends to the origin server when an object cannot be served from cache. The bereq object contains the necessary backend request information and is built from the req object.

In terms of scope, the req object is accessible in client-side subroutines, whereas the bereq object is only accessible in backend subroutines.

Although both objects are quite similar, they are not identical. The backend requests do not contain the per-hop fields such as the Connection header and the Range header.

Here are some bereq variables:

• bereq.url is the backend request URL.
• bereq.method is the backend request method.
• bereq.http.* contains the backend request headers.
• bereq.proto is the backend request protocol that was used.

On the one hand, bereq provides a copy of the client request information in a backend context. But on the other hand, because the backend request was initiated by Varnish, we have a lot more information in bereq:

bereq.connect_timeout, bereq.first_byte_timeout, and bereq.between_bytes_timeout contain the timeouts that are applied to the backend.

bereq.is_bgfetch is a boolean that indicates whether or not the backend request is made in the background. When this variable is true, this means the client hit an object in grace, and a new copy is fetched in the background.

bereq.backend contains the backend that Varnish will attempt to fetch from. When it is used in a string context, we just get the backend name.

Backend response variables#

Where there’s a backend request, there is also a backend response.

The beresp object contains all the relevant information regarding the backend response.

• beresp.proto contains the protocol that was used for the backend response.
• beresp.status is the HTTP status code that was returned by the origin.
• beresp.reason is the HTTP status message that was returned by the origin.
• beresp.body contains the response body, which can be modified for synthetic responses.
• beresp.http.* contains all response headers.

VFP-related backend response variables#

There are also a bunch of backend response variables that are related to the Varnish Fetch Processors (VFP). These are booleans that allow you to toggle certain features:

• beresp.do_esi can be used to enable ESI parsing.
• beresp.do_stream can be used to disable HTTP streaming.
• beresp.do_gzip can be used to explicitly compress non-gzip content.
• beresp.do_gunzip can be used to explicitly uncompress gzip content and store the plain text version in cache.

Timing-related backend response variables#

• beresp.ttl contains the objects remaining time to live (TTL) in seconds.
• beresp.age (read-only) contains the age of an object in seconds.
• beresp.grace is used to set the grace period of an object.
• beresp.keep is used to keep expired and out of grace objects around for conditional requests.

These variables return a duration type. beresp.age is read-only, but all the others can be set in the vcl_backend_response or vcl_backend_error subroutines.

Other backend response variables#

• beresp.was_304 indicates whether or not our conditional fetch got an HTTP 304 response before being turned into an HTTP 200.
• beresp.uncacheable is inherited from bereq.uncacheable and is used to flag objects as uncacheable. This results in a hit-for-miss, or a hit-for-pass object being created.
• beresp.backend returns the backend that was used.

Object variables#

The term object refers to what is stored in cache. It’s read-only and is exposed in VCL via the obj object.

Here are some obj variables:

• obj.proto contains the HTTP protocol version that was used.
• obj.status stores the HTTP status code that was used in the response.
• obj.reason contains the HTTP reason phrase from the response.
• obj.hits is a hit counter. If the counter is 0 by the time vcl_deliver is reached, we’re dealing with a cache miss.
• obj.http.* contains all HTTP headers that originated from the HTTP response.
• obj.ttl is the object’s remaining time to live in seconds.
• obj.age is the objects age in seconds.
• obj.grace is the grace period of the object in seconds.
• obj.keep is the amount of time an expired and out of grace object will be kept in cache for conditional requests.
• obj.uncacheable determines whether or not the cached object is uncacheable.

Response variables#

The resp object contains the necessary information about the response that is going to be returned to the client.

In case of a cache hit the resp object is populated from the obj object. For a cache miss or if the cache was bypassed, the resp object is populated by the beresp object.

When a synthetic response is created, the resp object is populated from synth.

Here are some response variables as an example:

• resp.proto contains the protocol that was used to generate the HTTP response.
• resp.status is the HTTP status code for the response.
• resp.reason is the HTTP reason phrase for the response.
• resp.http.* contains the HTTP response headers for the response.
• resp.is_streaming indicates whether or not the response is being streamed while being fetched from the backend.
• resp.body can be used to produce a synthetic response body in vcl_synth.

Storage variables#

In varnishd you can specify one or more storage backends, or stevedores as we call them. The -s runtime parameter is used to indicate where objects will be stored.

Here’s a typical example:

varnishd -a :80 -f /etc/varnish/default.vcl -s malloc,256M

This Varnish instance will store its objects in memory and will allocate a maximum amount of 256 MB.

In VCL you can use the storage object to retrieve the free space and the used space of stevedore.

In this case you’d use storage.s0.free_space to get the free space, and storage.s0.used_space to get the used space.

When a stevedore is unnamed, Varnish uses the s%d naming scheme. In our case the stevedore is named s0.

Let’s throw in an example of a named stevedore:

varnishd -a :80 -f /etc/varnish/default.vcl -s memory=malloc,256M

To get the free space and used space, you’ll have to use storage.memory.free_space and storage.memory.used_space.

The built-in VCL#

The built-in VCL is a collection of VCL code that will be executed by default, even if the code is not specified in your VCL file. The built-in VCL is defined in the various VCL subroutines and is called by the Varnish Finite State Machine.

It is possible to bypass the built-in VCL by issuing a return statement in the corresponding subroutine.

To learn more about Varnish’s built-in VCL, take a look at the corresponding tutorials.