SELinux FAQ

What is SELinux?#

SELinux stands for Security-Enhanced Linux. It’s a Linux Security Module (LSM) that enables permissions and restrictions with a much finer granularity than traditional Unix permissions.

In short, SELinux works by labeling resources with types and establishing permissions for certain operations between types. For example, a process running the program /usr/sbin/varnishd has the type varnishd_t and the files inside the /etc/varnish directory have the type varnishd_etc_t. There exists a rule to allow varnishd_t to read varnishd_etc_t files, for example /etc/varnish/default.vcl. SELinux also has its own flavor of users and roles and comes with a significant learning curve.

Once the varnish service is started, you should be able to observe two processes:

$ ps -C varnishd,cache-main -o comm,label
COMMAND         LABEL
varnishd        system_u:system_r:varnishd_t:s0
cache-main      system_u:system_r:varnishd_t:s0

The label gives us the security context for /usr/sbin/varnishd:

• system_u is the built-in system user meant for system services
• system_r is the built-in system role
• varnishd_t is a type
• s0 is a range

The important part in this case is that varnishd is labeled with the varnishd_t type. Even though varnishd is executed as the root user, SELinux will deny any operation that a root user could normally perform if that operation is not allowed for the varnishd_t type.

How do I disable it?#

It used to be almost systematic to solve a permission problem by disabling SELinux and you should avoid that. What SELinux solves is the loophole of the root user: in theory the root user (UID 0) is the privileged user and can do anything. For instance, port numbers below 1024 are privileged ports, so in order to simply run a standard web server on ports 443 and 80 you need root privileges.

Varnish was designed with security in mind, and the workload is split in two processes:

• The management process operating in the control plane
• The cache process operating in the data plane

With its jail facility Varnish will permanently drop privileges in the cache process, and temporarily drop them in the management process to only reacquire privileges with the smallest possible window to perform privileged operations.

While Varnish proactively tries to mitigate many attack vectors, what happens when a severe security vulnerability is discovered in any piece of software? If a remote code execution can be achieved on a process running as root then an attacker can do virtually anything. SELinux defends against that and will for example prevent Varnish from reading your mail server configuration if you happen to run both services on the same system. SELinux comes with a policy covering a large part of critical system software like Varnish and can greatly reduce the blast radius of certain classes of vulnerabilities.

What is enforcing mode?#

When SELinux is enabled it can essentially run in two modes:

• Enforcing mode
• Permissive mode

In enforcing mode, anything denied by the policy will be blocked and recorded in the system’s audit log. The application will likely see denied operations fail with operation not permitted (EPERM), and how this error is handled is up to the application.

In permissive mode, an unauthorized operation is only logged, not denied, which allows the application to keep making progress while informing system administrators that the policy would have denied certain operations.

How to troubleshoot SELinux?#

Ideally, running a comprehensive workload in permissive mode should allow you to catch most if not all the things that would go wrong in enforcing mode.

You can then search the audit log with ausearch and even generate SELinux policies with audit2allow. When working with Varnish Enterprise, the best way to help us study SELinux denials is to send us a varnishgather report after running your workload in permissive mode.

See varnishgather on GitHub.

You don’t have to jeopardize the security of your entire system by putting it in permissive mode. Instead, you can put a specific type from the policy in permissive mode:

semanage permissive -a varnishd_t # add to the list of permissive types
semanage permissive -d varnishd_t # delete permissive type
semanage permissive -l            # list permissive types

As mentioned earlier, SELinux will prevent Varnish from reading your mail server configuration. This works by abstracting resources, such as paths, programs or port numbers with types, and then allowing certain types to perform specific operations on other types.

The SELinux policy contains several types related to Varnish, you may put in permissive mode the one that is denied operations according to the audit log.

It will very likely be one of the following:

• varnishd_t
• varnishlog_t (this includes varnishncsa)

For example, varnishd_t is allowed to read configuration files labeled as varnishd_etc_t. Some utilities like ls can take a -Z option to print the security context of files and directories:

$ ls -Z /etc/varnish/*.vcl
unconfined_u:object_r:user_home_t:s0 /etc/varnish/custom.vcl
 system_u:object_r:varnishd_etc_t:s0 /etc/varnish/default.vcl

If you observe files with the wrong security context, this can be corrected with the restorecon utility:

$ restorecon -v -r /etc/varnish
Relabeled /etc/varnish/custom.vcl from unconfined_u:object_r:user_home_t:s0 to unconfined_u:object_r:varnishd_etc_t:s0

If your SELinux troubleshooting reveals something more complicated than an unexpected label on a file, remember to contact our support if you are a Varnish Enterprise customer. For the quickest possible resolution, include a varnishgather report after running your workload with the problematic type in permissive mode.

Who manages the SELinux policy?#

Varnish Enterprise supports SELinux on Red Hat Enterprise Linux (RHEL) and compatible systems. The policy is developed and maintained by Red Hat, and we at Varnish Software contribute fixes and enhancements to the policy when we update our downstream policy.

What does the varnishd policy do?#

SELinux is modular, so there is a base policy and a set of modules on top of that. There is a varnishd semodule covering several components from Varnish Cache.

In broad strokes, the varnishd program is allowed to do the following:

• Listen to HTTP ports
• Connect to HTTP ports
• Optionally connect to any port via a boolean called varnishd_connect_any
• It has full access to its working directory in the /var/lib/varnish tree
• It can load VCL from /etc/varnish

It also allows varnishlog and varnishncsa to access Varnish’s shared memory log and persist logs in /var/log/varnish.

Why implement a varnish-plus policy?#

On systems where we support SELinux we disable the varnishd semodule and add our own varnish-plus semodule instead. We do this because Varnish Enterprise has more features and some of them would otherwise be denied by default.

We could have added a complementary semodule and leave the varnishd one alone, but we support both RHEL7 and RHEL8 and their policies target respectively Varnish 4.0 and 6.0 between which there is a noticeable gap.

It was much simpler to maintain a single policy for both platforms. By effectively forking the varnishd semodule we also took care of isolating Enterprise-only permissions, which gives us a better understanding of whether a change we are making is relevant for the upstream policy and should be contributed back.

The one noticeable departure from the upstream policy is the split of the varnishd_port_t in two:

• varnishd_http_port_t (data plane)
• varnishd_cli_port_t (control plane)

How can I grant my third-party VMOD extra permissions?#

All VMODs are running inside varnishd, and are subject to the same SELinux policy. Even our vmod_memcached would likely be denied a connection to a Memcached server based on Memcached’s default port number.

There are several ways to deal with this. You could add the Memcached port to the varnishd_http_port_t directly from a command line with the semanage utility. Or you could create your own semodule to grant varnishd_t access to memcache_port_t.

Unless you are proficient enough with SELinux to extend the policy, you could simply turn a built-in boolean on:

semanage boolean --modify --on varnishd_connect_any

When in doubt, contact Varnish support.

Why is SELinux preventing a package upgrade?#

When we first introduced our downstream varnish-plus policy, it needed to disable an existing varnish SELinux module. Unfortunately, it didn’t handle all scenarios properly. In some cases, it could fail to install the varnish-plus SELinux module and allow the varnish-plus package installation to proceed.

Using off-the-shelf RPM macros to manage SELinux modules solved the original issue. This approach resulted in generally better integration, especially during upgrades, but it still wouldn’t prevent a broken package installation from proceeding. At that time we did not realize that the RPM macros use a different priority level than our initial integration.

The introduction of RHEL8 was a regression from the previous simplification. Our former varnish-plus-ports SELinux module was installed as CIL code by the RPM macros. This code would no longer compile on RHEL8, forcing us to manage varnishd_http_port_t and varnishd_cli_port_t types manually.

Next, the introduction of Amazon Linux 2 brought more complications to the process. It advertises itself as RHEL7 in its RPM setup, but has outdated dependencies, such as SELinux. Because of this, we have had to entirely circumvent our SELinux policy on this platform.

As a result, a series of bug fixes have been implemented to remediate all known failure scenarios and prevent a package installation when a failure would occur. This could result in the following error messages:

Error: could not disable the varnishd SELinux module.
error: %pre(varnish-plus-selinux-X.Y.ZrT-1.elV.noarch) scriptlet failed, exit status 1
Error in PREIN scriptlet in rpm

This can occur when there is a dependency to the varnishd SELinux module in the system. For example, when the varnishd_t type is set to be permissive, which under the hood creates a permissive_varnishd_t SELinux module.

Or if in order to solve a denial, for example to allow monitoring software to access something owned by Varnish, applying a policy update that was suggested by the audit logs tooling can also take the form of an SELinux module.

In those scenarios you can temporarily disable the offending module:

$ semodule -d permissive_varnishd_t
# perform package update
$ semodule -e permissive_varnishd_t

This should be a one-time workaround, because once the varnishd SELinux module is properly replaced by our own varnish-plus policy, upgrades can happen safely even when a module depends on it.

When in doubt, contact our support.

Why can’t varnishd load my VCL file?#

One frequently recurring mistake is to copy files via SSH and move them to the /etc/varnish directory. Whether it is done manually or through automation, it should be noted that when a file is effectively moved, it will keep the security context it inherited when it was created.

So it is common that VCL files in /etc/varnish end up with the user_home_t type. You can fix this as part of your deployment by adding a restorecon step or by updating the security context during the move:

$ ls -Z custom.vcl
unconfined_u:object_r:user_home_t:s0 custom.vcl
$ mv -v -Z custom.vcl /etc/varnish
copied 'custom.vcl' -> '/etc/varnish/custom.vcl'
removed 'custom.vcl'
$ ls -Z /etc/varnish/custom.vcl
unconfined_u:object_r:varnishd_etc_t:s0 /etc/varnish/custom.vcl

There are other utilities like ps that can take a -Z option, not just ls and mv.

Why is varnishncsa denied write access to logs?#

Our packaging comes with a logrotate configuration, and the idea is to always append access logs and rotate them to avoid overwriting and losing logs on disk as we are building them up. The lack of arbitrary write permission comes from the upstream varnishd semodule and it looks sensible enough that we decided to keep it that way.