Using the PROXY protocol in Varnish

The PROXY protocol was developed by HAProxy to let proxy servers communicate meta information to servers before starting the normal proxying behavior. Its initial version focused on transmitting basic information on the forwarded stream, but the protocol was later expanded to contain additional information.

Varnish can be configured to accept incoming connections over the PROXY protocol and processes the PROXY header before processing the rest of the incoming payload as regular HTTP. The information from the PROXY header is used to populate VCL variables, such as client.ip and server.ip, and to modify the X-Forwarded-For request header.

The PROXY protocol header#

There are two versions of the PROXY protocol:

• In version 1 the PROXY header is sent in plain text and contains limited client connection information.
• In version 2 the PROXY header is sent in binary format and extends the set of information that can be sent using the protocol.

PROXY protocol version 1#

Version 1 of the PROXY protocol consists of a plain text header that is prepended to the HTTP request as you can see in the example below:

PROXY TCP4 10.10.10.1 10.10.10.2 58076 443
GET / HTTP/1.1
Host: example.com

The header starts with PROXY and has the following fields:

• TCP4: the proxied INET protocol and family. This could also be TCP6.
• 10.10.10.1: the source addresss. This could also be an IPv6 address.
• 10.10.10.2 : the destination address. This could also be an IPv6 address.
• 58076 : the TCP source port
• 443: the TCP destination port

PROXY protocol version 2#

Version 2 of the PROXY protocol is in a binary format that contains a lot more connection information than version 1. It also counters a lot of the limitations of the first version.

Whereas version 1 only supported TCP connections over IPv4 and IPv6, version 2 supports:

• TCP over IPv4
• TCP over IPv6
• UNIX streams over UNIX domain sockets (UDS)
• UDP over IPv4
• UDP over IPv6
• UNIX datagrams over UNIX domain sockets (UDS)

TLV attributes#

Version 2 of the PROXY protocol also contains optional TLS-related TLV attributes in case the connection was made over TLS.

The following TLV attributes are available:

• The Application-Layer Protocol Negotiation (ALPN) attribute
• The Authority attribute, which is the hostname that was used for the TLS connection
• The CRC32c attribute, which is the checksum of the PROXY header
• The unique ID attribute that identifies the connection
• The client SSL flag attribute that indicates whether or not a TLS/SSL connection was used
• The client certificate connection flag attribute that indicates whether or not a certificate was presented over the current connection
• The client certificate session flag attribute that indicates whether or not a certificate was presented over the TLS session the connection belongs to
• The client verification flag attribute that indicates whether or not the presented certificate was successfully verified
• The SSL cipher attribute that lists the encryption ciphers that were negotiated
• The SSL signing algorithm attribute that indicates which algorithm was used to sign the certificate
• The SSL key algorithm attribute that indicates which algorithm was used to generate the private key of the certificate
• The NETNS attribute that defines the namespace that was used

Use case: TLS termination#

The following diagram illustrates how the PROXY protocol can be used to facilitate TLS termination:

Here’s what happens:

• The client (10.10.10.1) connects to server 10.10.10.2 on port 443 to send an HTTPS request.
• This server (10.10.10.2) is a TLS proxy, terminates the TLS session and forwards the decrypted request data to Varnish (10.10.10.3).
• Varnish is configured to receive incoming PROXY protocol requests on port 8843.
• Varnish receives an incoming request from 10.10.10.2 and based on the regular connection information it thinks this is the client.
• In reality 10.10.10.1 is the client and the PROXY protocol header is able to transport this information to Varnish.
• Varnish is then able to set X-Forwarded-For: 10.10.10.1 to inform the origin server (10.10.10.4) who the actual client is.

Enabling the PROXY protocol in Varnish#

You can enable the PROXY protocol in Varnish by configuring a listening address that uses PROXY as the protocol. The listening addresses are configured through the -a runtime parameter of the varnishd program.

Here’s a standard HTTP implementation without PROXY support:

varnishd -a :80 -f /etc/varnish/default.vcl

This example will register port 80 as the listening port and because no protocol was specified, HTTP will be used.

This is the equivalent of the following configuration:

varnishd -a :80,HTTP -f /etc/varnish/default.vcl

Through the PROXY keyword we will now enable PROXY support in Varnish. You can either enable it over a regular TCP/IP or over a UNIX domain socket.

PROXY protocol over TCP#

To enable the PROXY protocol, simply add another listening address to the varnishd runtime configuration:

varnishd -a :80 -a :8443,PROXY -f /etc/varnish/default.vcl

Then it’s just a matter of connecting the first upstream proxy to Varnish over port 8443 and ensure the proxy protocol is used for backend communication on that proxy server.

PROXY protocol over UNIX domain sockets#

If you are hosting another proxy on the same server as Varnish, you don’t have to use TCP. For local connections, you can use a UNIX domain socket.

Although it requires a bit more configuration, UDS takes away some of the overhead and is ideal for TLS proxies that process a lot of traffic.

Here’s an example of a listening address that uses a UNIX domain socket for incoming connections:

varnishd -a :80 \
 -a /var/run/varnish.sock,PROXY,user=varnish,group=varnish,mode=660 \
 -f /etc/varnish/default.vcl

Incoming requests are read from the /var/run/varnish.sock socket, so make sure that file exists. The file is owned by the varnish user and by the varnish group. Both the user and the group are allowed to read from the socket and write to the socket.

Systemd configuration#

In a lot of cases, your production system will use systemd to manage the varnishd program and its runtime parameters.

If you want to enable the PROXY protocol on a systemd managed system, run the following command to edit and override the unit file:

sudo systemctl edit varnish

An editor will open. Here’s an example of what you can add to this override to enable the PROXY protocol:

[Service]
ExecStart=
ExecStart=/usr/sbin/varnishd \
	  -a :80 \
	  -a localhost:8443,PROXY \
	  -p feature=+http2 \
	  -f /etc/varnish/default.vcl \
	  -s malloc,2g

Restart Varnish to effectuate the changes and to enable the PROXY protocol:

sudo systemctl restart varnish

Our example systemd configuration has the following listening address for PROXY traffic:

-a localhost:8443,PROXY

This listening address will only be available for local traffic on port 8443. This is ideal when you host your TLS proxy on the same machine as Varnish.

An alternative configuration for local connections would be through a UNIX domain socket:

-a /var/run/varnish.sock,PROXY,user=varnish,group=varnish,mode=660

PROXY traffic on a regular HTTP interface?#

If you haven’t enabled the PROXY protocol on your Varnish server, but the proxy server in front of Varnish is sending PROXY traffic, the output of varnishlog -g session will tell you that something is wrong.

Here’s what you see in the logs when you use version 1 of the PROXY protocol:

*   << Session  >> 23
-   Begin          sess 0 HTTP/1
-   SessOpen       127.0.0.1 36516 a0 127.0.0.1 80 1642420246.277563 27
-   Link           req 24 rxreq
-   SessClose      RX_JUNK 0.000
-   End
**  << Request  >> 24
--  Begin          req 23 rxreq
--  Timestamp      Start: 1642420246.277603 0.000000 0.000000
--  Timestamp      Req: 1642420246.277603 0.000000 0.000000
--  BogoHeader     Illegal char 0x20 in header name
--  HttpGarbage    "PROXY%00"
--  RespProtocol   HTTP/1.1
--  RespStatus     400
--  RespReason     Bad Request
--  ReqAcct        114 0 114 28 0 28
--  End

You can see that Varnish receives the plain text PROXY header, but doesn’t consider it to be valid HTTP. The -- BogoHeader Illegal char 0x20 in header name log line sees the input as bogus HTTP. The -- HttpGarbage "PROXY%00" log line doesn’t expect an HTTP request to start with PROXY and considers it garbage.

When Varnish receives such a request, it will return an HTTP/1.1 400 Bad Request error, which is also reflected in the logs.

If you use version 2 of the PROXY protocol to connect to a regular HTTP interface, this is what will appear in the logs:

*   << Session  >> 20
-   Begin          sess 0 HTTP/1
-   SessOpen       127.0.0.1 36512 a0 127.0.0.1 80 1642420185.334654 30
-   Link           req 21 rxreq
-   SessClose      RX_JUNK 0.000
-   End
**  << Request  >> 21
--  Begin          req 20 rxreq
--  Timestamp      Start: 1642420185.334701 0.000000 0.000000
--  Timestamp      Req: 1642420185.334701 0.000000 0.000000
--  HttpGarbage    "%0d%0a%0d%0a%00"
--  RespProtocol   HTTP/1.1
--  RespStatus     400
--  RespReason     Bad Request
--  ReqAcct        156 0 156 28 0 28
--  End

Because the content is not in plain text format, there will be no BogoHeader line in the logs, but the binary content is considered invalid. The -- HttpGarbage "%0d%0a%0d%0a%00" log line reflects this.

In this scenario an HTTP/1.1 400 Bad Request error will also be returned to the client.

HTTP traffic on a PROXY interface?#

If you enabled the PROXY protocol on your Varnish server, but the proxy server in front of Varnish uses regular HTTP, the output of varnishlog -g session will be the following:

*   << Session  >> 13
-   Begin          sess 0 PROXY
-   SessOpen       127.0.0.1 39696 a1 127.0.0.1 8443 1642418713.617643 26
-   SessClose      RX_JUNK 0.000
-   End

You can see that the logs indicate - Begin sess 0 PROXY. This means that the listening address that was used expects PROXY traffic. Since that is not the case, the connection is closed immediately. The RX_JUNK keyword hints that Varnish received junk content.

PROXY information in the logs#

When a valid connection using the PROXY protocol has been set up, the Proxy tag will appear in the logs.

It displays the following fields:

• Protocol version
• Source address
• Source port
• Destination address
• Destination port

When you run varnishlog -g session -i Proxy, you can receive the following output:

-   Proxy          1 10.10.10.1 58076 10.10.10.2 443

This example features version 1 of the proxy protocol. The original client IP address is 10.10.10.1 that used port 58076 on its system to connect to server 10.10.10.2 on port 443.

Here’s the output from the varnishlog -g session -i SessOpen -i Proxy command that displays the session connection information from Varnish as well as the proxy information from the original client connection:

*   << Session  >> 33
-   SessOpen       10.10.10.2 36670 a1 10.10.10.3 8443 1642421783.179568 23
-   Proxy          1 10.10.10.1 58076 10.10.10.2 443
**  << Request  >> 34

The SessOpen tag shows that Varnish accepted a session on port 8443 from a client that operates on the local machine using port 36670 . The Proxy tags show that the actual client is 10.10.10.1. This client connected to IP address 10.10.10.2 on port 443.

If you’re using version 2 of the proxy protocol, the only difference is that the first field is 2 instead of 1.

-   Proxy          2 10.10.10.1 58076 10.10.10.2 443

vmod_proxy#

By accepting PROXY protocol traffic in Varnish some variables will be set automatically, such asclient.ip and server.ip. The value of the X-Forwarded-For header can also contain the original client IP address.

However, to retrieve the TLV attributes from a Proxy v2 header, you need to use vmod_proxy.

This Varnish module is part of the standard Varnish installation and can be imported in your VCL file through import proxy;.

Its API goes as follows:

STRING alpn()

STRING authority()

BOOL is_ssl()

BOOL client_has_cert_sess()

BOOL client_has_cert_conn()

INT ssl_verify_result()

STRING ssl_version()

STRING client_cert_cn()

STRING ssl_cipher()

STRING cert_sign()

STRING cert_key()

These functions primarily return TLS-related information that helps you verify the validity of the TLS session, but also return certificate and protocol information.

Logging TLS/SSL versions#

An example use of vmod_proxy is logging the TLS/SSL version that was used:

vcl 4.1;

import std;
import proxy;

backend default {
    .host = "127.0.0.1";
    .port = "8080";
}

sub vcl_recv {
    std.log("TLS-SSL-VERSION: " + proxy.ssl_version());
}

The varnishlog -g session -i Proxy -I VCL_Log:TLS-SSL-VERSION will display the PROXY header information as well as the TLS/SSL version.

Here’s the output:

*   << Session  >> 31
-   Proxy          2 10.10.10.1 58076 10.10.10.2 443
**  << Request  >> 32
--  VCL_Log        TLS-SSL-VERSION: TLSv1.2
*** << BeReq    >> 32773

We can conclude that the client identified by IP address 10.10.10.1 used TLSv1.2.

Setting the X-Forwarded-Proto header based on the PROXY header#

Because the connection between Varnish and the origin web server is made over plain HTTP, the application might force an HTTPS redirection because it assumes the original connection was made using HTTP.

Fortunately a lot of applications support the X-Forwarded-Proto header. It contains the original request protocol and should be set by the first proxy server in the chain.

If that proxy didn’t set the header, or if the proxy is not an HTTP proxy, we can use proxy.is_ssl() to determine the value and set the X-Forwarded-Proto header in Varnish.

Here’s the VCL code you need:

vcl 4.1;

import proxy;

backend default {
    .host = "127.0.0.1";
    .port = "8080";
}

sub vcl_recv {
    if(!req.http.X-Forwarded-Proto) {
        if (proxy.is_ssl()) {
            set req.http.X-Forwarded-Proto = "https";
        } else {
            set req.http.X-Forwarded-Proto = "http";
        }
    }    
}

If a plain HTTP request was made, the value of the X-Forwarded-Proto header will be http. If it was an HTTPS request, the value will be https.