Running Varnish Enterprise 6 for Ubuntu on AWS

When running Varnish on Amazon Web Services, you can leverage Varnish Software’s official machine images on the AWS marketplace.

In this tutorial we’ll cover the Varnish Enterprise 6 for Ubuntu image. When you spin up an EC2 instance using this AMI, you get a pre-installed Varnish setup that is configured to meet Varnish Software’s best practices.

1. Launch instance#

Click the Launch instances button on the EC2 overview of your AWS console.

• Instead of choosing one of the Quick Start images, click on the AWS Marketplace
• In the search box you can type varnish and press return to see the results
• The Varnish Enterprise 6 (Ubuntu) image should be the second result in the list
• Click the Select button next to the image to proceed
• You will get an overview of the image details and pricing
• Click Continue

2. Choose the instance type#

Choosing the right instance type has a direct impact on the amount of objects you can cache in Varnish.

For standard use cases properties like vCPUs and Instance Storage is not that important. Unless you use Varnish Enterprise’s Massive Storage Engine with disk persistence, disk storage is irrelevant. Varnish is also not that CPU intensive.

The properties that do matter are Memory and Network Performance.

Assume that 80% of your server’s memory will be assigned to Varnish. Depending on your traffic patterns and VCL configuration, the vast majority of this memory can be used for object storage. Your traffic patterns will also dictate the amount of network performance you need.

Click Next: Configure Instance Details after selecting the instance type to go to the next step.

3. Configure instance#

In the third step of the installation procedure, you can configure the instance:

• You can choose the number of instances you want to launch
• You can even choose to launch you instances in an autoscaling group
• You can choose your Virtual Private Network, as well as the IP subnet
• You can configure the hostname and DNS name of the server
• There’s a variety of options to configure, but in a lot of cases the default values will do
• Click Next: Add Storage to go to the fourth step of the install wizard.

4. Add storage#

As mentioned earlier: disk storage is usually not a concern. Varnish stores objects in memory and either uses the malloc storage engine or mse storage engine for that.

However, if you’re planning to cache large volumes of data that go beyond the capacity of the memory, you can use the mse storage engine with extra configuration for disk persistence.

Click Next: Add Tags to go to the fifth step of the install wizard.

5. Add tags#

In the fifth step of the install wizard of our Varnish EC2 instance, we can assign tags to the instance. This is optional, but we would advise you to assign a Name tag.

Click Next: Configure Security Group to go to the next step.

6. Configure Security Group#

A Security Group is AWS' interpretation of a firewall configuration. The AMI has some security group presets and lists the ports that should be opened.

These range from port 22 for SSH, to 80 and 443 for web access. Additional ports such as 6081, 6085, 8088 and 8089 are also opened. These ports are there for management purposes.

Unless you want to customize the firewall rules, click Review and Launch to continue.

7. Review#

In the final step of the install wizard, you can review all the settings of the Varnish server you’re about to launch.

Click the Launch button to continue. A popup will appear asking which SSH key pair can be used to access the server. If you already have a keep pair, you can select it in the dropdown list.

In the tutorial video we chose to create a new key pair which we downloaded to our computer. The public key will be registered on the server, the private key can be used to access the machine via SSH.

Click Launch Instances to start the creation and provisioning of your Varnish server. After a couple of minutes the server will be ready for action.

8. Preview#

• Go back to your EC2 instances overview
• Select your newly created Varnish server
• Click the copy icon next to the public IP address in the Details tab of the instance
• Paste the IP address in a browser window to preview your Varnish server.

The landing page is generated by Varnish and has a call to action button that links to an instruction guide.

In the next steps we will access the server via SSH an edit some of the configuration settings to ensure Varnish properly caches the content from your web server.

9. SSH access#

To access the server via SSH, you need the following information:

• The IP address of your EC2 instance
• The username to access the machine
• An SSH private key to authenticate

The following example will access server 18.133.187.177. The username for the Ubuntu version of Varnish Enterprise 6 on AWS is ubuntu. During the setup, we created and downloaded the varnish.pem key pair. We’ll use this key pair to authenticate:

ssh -i varnish.pem ubuntu@18.133.187.177

10. Basic VCL changes#

Now that we have access to the server, the default VCL file should be modified. It should contain the IP address of the web server and the VCL call that displays the landing page should be removed.

Open the /etc/varnish/default.vcl file with your editor of choice. We use vim:

sudo vim /etc/varnish/default.vcl

Update the .host and .port properties of the backend and use the IP address of the webserver and port 80. In our example that the web server’s IP address is 18.170.69.4:

vcl 4.1;

backend default {
    .host = "18.170.69.4";
    .port = "80";
}

To make sure the landing page is no longer displayed, remove the return(synth(200,"Welcome")); line from the vcl_recv subroutine.

Run the following command to restart Varnish and reload the VCL:

sudo systemctl restart varnish

11. Modify the object storage#

In the previous step we changed the VCL file, but that’s not the only part of the configuration you can modify. The varnishd program also has a collection of runtime parameters that can be overrided.

Run the following command to edit the varnishd runtime parameters for the Varnish systemd service:

sudo systemctl edit varnish

An editor will open with the following content:

[Service]

ExecStart=
ExecStart=/usr/sbin/varnishd -a :80 -a 127.0.0.1:8443,proxy \
 -S /etc/varnish/secret -T localhost:6082 \
 -f /etc/varnish/default.vcl \
 -s malloc,256m -p syslog_cli_traffic=off

malloc storage#

The -s runtime parameter defines the storage engine. The standard storage engine is malloc, which is a memory-only storage engine that is also available in the open source version. The default storage size is 256m which is very limited.

You can increase the value to about 80% of the total memory of the server.

Save and close the editor and run the following command to restart Varnish with the updated runtime parameters:

sudo systemctl restart varnish

Massive Storage Engine#

Since this server runs Varnish Enterprise, you can also use the Massive Storage Engine (MSE).

Not only does MSE combine memory and persisted disk storage, it also has a feature called Memory Governor. This feature will allow the varnishd process to be limited in size, rather then the object storage itself.

Thanks to the Memory Governor Varnish will keep a stable memory footprint while scaling the size of the cache based on available memory.

By replacing -s malloc,256m with -s mse, MSE will be active with Memory Governor enabled. The memory_target runtime parameter defines the total memory consumption of the varnishd program and has 80% as its default value.

Save and close the editor and run the following command to restart Varnish with the updated runtime parameters:

sudo systemctl restart varnish

12. TLS configuration#

The Varnish Enterprise 6 for Ubuntu image is equiped with Hitch, a TLS proxy that handles TLS termination.

The TLS configuration is stored in /etc/hitch/hitch.conf and managed by the hitch systemd service.

Our Hitch tutorial explains all the details, so we recommend that you take a look at it.

Self-signed certificates#

But for now, the only configuration directive that really matters is the pem-file directive. It refers to /etc/hitch/testcert.pem. This file is the self-signed certificate that is loaded.

If you want to test HTTPS on the server locally, you can run the following command:

curl -k https://localhost

The -k option will ignore verification of the certificate. This ensures that a TLS session can be established using the self-signed certificate.

CA certificates#

For production traffic we cannot use self-signed certificates. It is important that you upload your TLS certificates to the server and that you register them in Hitch.

You can either overwrite the /etc/hitch/testcert.pem file with your actual TLS certificate, or you can write the certificate to another filename and change the pem-file directive.

If you’re planning to use multiple certificates, you can use the following configuration lines in /etc/hitch/hitch.conf:

pem-dir = "/etc/hitch/cert.d"
pem-dir-glob = "*.pem"
pem-file = "/etc/hitch/testcert.pem"

This configuration will load all the certificates from the /etc/hitch/cert.d folder. Valid certificate files end with the *.pem extension. Server Name Indication (SNI) support in Hitch will ensure the right certificate is matched based on the hostname.

If no certificate could be matched in the /etc/hitch/cert.d folder, the /etc/hitch/testcert.pem file will be used as a fallback.

Confirm TLS changes#

When you have successfully uploaded your certificates and modified the Hitch configuration, you should restart Hitch to ensure these changes take effect.

Run the following command to restart Hitch:

sudo systemctl restart hitch

13. Customizing the VCL#

The VCL code that is used on this server is largely based on the built-in VCL. This is VCL code that is executed in the various VCL subroutines by the Varnish finite state machine. It is possible to bypass the built-in VCL by issuing a return statement in your VCL code.

The built-in VCL is very conservative and prevents personalized content from ending up in the cache. When the built-in VCL notices a Set-Cookie response header it will not store that response in the cache. When it notices a Cookie request header it will not serve the corresponding object from the cache.

For real-world applications, the built-in VCL will probably not suffice. You will have to write some custom VCL code to ensure that your web platform is properly cached.

If you’re using a framework like WordPress, Drupal or Magento, you’ll find VCL templates in the integrations tutorial category.

Alternatively, you can also use our example VCL template that is not customized for any framework.

And if you need any more inspiration while customizing your VCL code, you can browse through our VCL tutorials.