The Long Road to SELinux, Part 3: Living with it

Table of Contents

By the end of 2003 SELinux was in the mainline Linux kernel, and if you had built 2.6.0 with it turned on, taken the NSA’s example policy (the kernel ships none), added our pipeline’s types and rules to its source tree and rebuilt the whole thing, your machine would have enforced our pipeline against every program except the ones Alice runs from her own login, and refused to do a great deal else. The policy the NSA had shipped, which everyone came to call the strict policy, was an example policy in the literal sense, a demonstration that the mechanism worked, not something you were supposed to run as is. Every daemon, every user and every shell was confined to a domain with exactly the permissions its author had thought of, and there was no way its author could have thought of everything your machine does. Printing broke, X broke, whatever your distribution’s init scripts did that the policy didn’t expect broke, and every one of those breaks produced an audit message that almost nobody could read.

From here on the story is about who writes the policy. Red Hat made SELinux usable by confining only what talks to the network and leaving the user alone, Android confined everything because one company writes the core policy and nobody holding the phone ever touches it, and most of the rest of the Linux world chose differently, names instead of labels and the developer instead of the platform. What forty years bought, in the end, is a check every Linux kernel can make on every access and a policy that only platforms reliably write.

Making it usable

So the mechanism was done and nobody could use it, and the next several years were spent by a fairly small number of people, most of them at Red Hat, turning it into something an administrator could live with. These are the years when SELinux earned its reputation for being difficult, and when most of what earned it got fixed, even if the reputation never caught up, and every one of these fixes assumes an administrator, which is exactly what a phone does not have, and what breaks when SELinux goes onto one.

The targeted policy

Fedora was the proving ground. Fedora Core 2, in the spring of 2004, had SELinux installed and switched off, you had to type selinux at the boot prompt to get it, because with the strict policy on too many things didn’t work. The fix that made Fedora Core 3 ship with SELinux on by default that November, and Red Hat Enterprise Linux 4 ship with it the following February, wasn’t really a technical change, it was the Red Hat team asking a different question.

The strict policy asked what every process on the machine is allowed to do, which is a question you can only answer if you know everything that will ever run on it. The targeted policy, which the Red Hat team (with Dan Walsh as its public face) put together for FC3, asked a much smaller question, which processes on this machine are actually worth confining? The answer they settled on was the ones that talk to the network, since those are the ones an attacker gets to talk to first. The web server, the name server, the DHCP server, the time daemon, the port mapper, the system logger, a dozen or so daemons that accept input from strangers (nine in the first Fedora Core 3 lists, twelve by RHEL 4, a couple of hundred domains by RHEL 5), each got a domain and a carefully written set of rules. Everything else, including every logged in user, root included, and everything they run, was put in one domain, unconfined_t, which is allowed to do anything the ordinary Unix permissions allow, which is to say SELinux gets out of the way. It works the other way too, a daemon that happens to run as root is still confined by its domain, the thing that stops httpd_t is the table, not the uid.

The targeted policy: users and most processes run unconfined, the daemons that talk to the network each get a confined domain, and booleans switch optional rules on and off unconfined_t SELinux gets out of the way Alice's shell and what she runs root's shell too most local programs can do whatever the Unix permissions allow, no more, no less confined domains the daemons that talk to the network httpd_t named_t dhcpd_t ntpd_t ... each with exactly the rules its author wrote, still confined when running as root booleans: httpd_can_network_connect = off Alice's plaintext, user_home_t (or plaintext_t with a local module) httpd_t has no rule for it, denied allowed, Alice is unconfined rule 2 holds against every confined daemon Rule 1 does not hold against Alice, because Alice is unconfined. That is the trade.
The targeted policy. Users and most programs are unconfined, the network facing daemons each get a domain, and the web server has no rule for Alice’s files no matter which user it runs as.

The objection from part 1 is obvious, this means Alice’s programs are back to being indistinguishable from one another, and it is true. But look at the bad day again. The attacker comes in through the web server, and the web server is httpd_t, and httpd_t has no access to Alice’s home directory, or to the mail spool, or to the name server’s zone files, or to anything that isn’t web content, and it does not matter that the attacker is root. The targeted policy gave up on least privilege for users in order to get least privilege for the things that actually get attacked, and that trade is why SELinux has been on by default on every Red Hat system for twenty years.

Reference policy, modules and booleans

The policy itself was rewritten at Tresys Technology in 2005 (Chris PeBenito, Frank Mayer and Karl MacMillan) as the reference policy, presented at the SELinux symposium in March 2006 and shipped in Fedora Core 5 that spring, and this is the base that Fedora, RHEL, Gentoo, Debian and everybody else builds from today. What it added over the NSA’s example was structure. Rules were grouped into modules, one per service, and modules only talked to each other through interfaces, small named macros that say things like this domain may read the web server’s content, so that the httpd module could be written without knowing anything about the internals of the mail module. And it was built to sit on a separate piece of Tresys work, Karl MacMillan’s loadable policy modules, presented at the symposium a year earlier, which is what let a distribution ship a hundred modules and an administrator load, unload or replace one of them without recompiling the rest.

That last part is a real change in the mechanism, not just in the text. With loadable policy modules and the semodule tool, the policy became something you assemble on the machine, the distribution’s modules plus whatever you wrote yourself, and the kernel’s security server loaded the result. Booleans were older, already in the Fedora Core 3 targeted policy and flipped with setsebool in RHEL 4, named switches inside the policy that the administrator can change at runtime without reloading it, and they are how the reference policy avoids having to decide questions it can’t know the answer to. Should the web server be allowed to make outgoing network connections? Some sites need that and some sites would consider it a compromise, so there is a boolean, httpd_can_network_connect, off by default, and the rules that allow it are wrapped in it. Should the web server serve pages out of users’ public_html directories? httpd_enable_homedirs, which lets it walk into home directories to reach the files there that the file contexts label as web content. Should it be able to read anything else a user keeps at home? httpd_read_user_content, off by default, and that is the one that would hand it Alice’s files. There are hundreds of these, and if you have ever fixed a SELinux problem with a single setsebool command, this is why.

Reading the denials

The other half of usability was explaining what had gone wrong. When SELinux denies something, the AVC writes a line to the audit log that looks like this

avc:  denied  { read } for  pid=2112 comm="httpd" name="secret.txt" dev=sda1 ino=8827
      scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file

Everything you need is there, who (httpd_t), what (a file of type user_home_t), which permission (read), but in 2004 almost nobody could read it, and the reflex that developed was to type setenforce 0, put SELinux into permissive mode (where it logs and does not deny), confirm that the thing now worked, and leave it there forever. Two tools were built against that reflex. audit2allow reads those denial lines and writes the allow rules that would have permitted them (an allow rule is one cell of the table from part 1 written as a line of policy, allow httpd_t user_home_t:file read), which is dangerous in exactly the way it sounds (you are teaching the policy to allow whatever just happened) and also, used with some judgement, the way most local policy modules got written. And setroubleshoot, which arrived in Fedora Core 6 in 2006, watches the audit log and turns each denial into a plain language explanation with a suggested fix, this is probably a mislabelled file, run this restorecon command, or if you really want the web server to read home directories, flip this boolean. If SELinux ever told you in a desktop notification that it is preventing something, that was setroubleshoot.

Labels on disk

Underneath all of this there is a plumbing question that the Flask paper had left to the object managers, where the labels actually live. On Linux the answer, from the 2.6 port on, is an extended attribute on every file, security.selinux, holding the context string, so that the label travels with the file rather than being looked up by path (the choice DTE had made, and that AppArmor would make again). Every file on the system has to have one, and it has to be the right one, and that is what file contexts are, a table in the policy that maps path patterns to the type a file at that path should have, /var/www/html(/.*)? is httpd_sys_content_t..etc. restorecon reads the table and sets the labels, the first boot after enabling SELinux relabels the whole filesystem, and a large fraction of all SELinux problems in the field come down to a file with the wrong label because it was created or moved by something that didn’t know the table, which is why the answer to so many questions is restorecon -R, and why semanage fcontext exists to add your own patterns.

Categories and levels

Two more things got bolted on in this period, and both matter later. The first is MCS, multi-category security, which arrived around Fedora Core 5 in 2006. It is the level field from the military models (that s0 at the end of the contexts above) with the hierarchy thrown away and only the compartments kept, so that a label can carry a set of categories, c1 and c42 say, and a process can only touch a file if its categories include the file’s. On its own that sounded like a way for users to mark their files, and almost nobody used it that way. The second is the real thing, an MLS policy, levels and all, which Red Hat Enterprise Linux 5 shipped in 2007 and which was evaluated that June, with IBM sponsoring and on IBM and HP hardware, at EAL4+ against the labelled security profile that descends from the Orange Book’s B1 class, the first Linux certified for the kind of labelled separation part 1’s multilevel security was about. Not A1, nothing general purpose has ever been, but the same family of policy, with the same trusted subjects and the same reasons nobody else wants it.

sVirt, and where the categories went

MCS found its purpose in 2009, when Dan Walsh and James Morris built sVirt into libvirt. The problem was virtual machines. Every guest on a host is a qemu process, and qemu is a large program that parses untrusted input from the guest, and if a guest escapes into its qemu process it is, under Unix permissions, the same user as every other qemu process on the host, with access to every other guest’s disk image. sVirt gives every guest’s qemu process the same type, svirt_t, and a randomly chosen pair of categories, and labels that guest’s disk images with the same pair, so that an escaped guest is confined to guest disk images in general by type enforcement, and to its own images only by MCS, without anyone writing a policy per guest.

sVirt: every guest's qemu process gets the same type and a random pair of categories, its disk images get the same pair, and an escaped guest cannot reach another guest's images guest 1, a qemu process svirt_t:s0:c12,c45 guest 1's disk image svirt_image_t:s0:c12,c45 same categories, allowed guest 2, compromised svirt_t:s0:c78,c90 guest 2's disk image svirt_image_t:s0:c78,c90 type enforcement says svirt_t may read svirt_image_t, the categories say only its own Nobody writes a policy per guest. libvirt picks the categories when the guest starts, and the same trick later kept containers apart.
sVirt. Type enforcement says a guest may touch guest images, the categories say only its own, and nobody has to write a rule per guest.

So it is the same type enforcement mechanism we have been following since part 1, with the MCS label added on top to tell the guests apart, and it closed off a very real attack surface. You have almost certainly used this without knowing. Docker arrived in 2013 without any of it, and in 2014 Red Hat’s work added the same thing (SELinux labels in Docker 0.11 that May, Dan Walsh’s –security-opt label options in 1.3 that October), every container process in one type, svirt_lxc_net_t then and container_t now, with a category pair per container. Podman does it, Kubernetes on those platforms does it, and a process that breaks out of a container’s namespaces on a Fedora or RHEL host, short of a kernel exploit, which no label stops, is still container_t with its own two categories, and cannot read Alice’s files or the other containers’ files, for exactly the reasons the web server in our example cannot read Alice’s plaintext.

Running the example, on a real machine

On a Fedora machine the example can finally be typed in. With the targeted policy, httpd runs as httpd_t, Alice’s files are user_home_t, and the web server can’t read them, out of the box, no work required, that is what the targeted policy is for. To get the rest of the pipeline you write a small module, a few lines declaring plaintext_t, ciphertext_t, encrypt_t and encrypt_exec_t, the cells of part 1’s table written out as allow rules, and a type_transition, build it, semodule -i it, add a file context for Alice’s plaintext directory with semanage fcontext, and restorecon it. Twenty lines and four commands, and the encrypt domain is the only confined domain that may read plaintext and write ciphertext, and the web server has no cell for either.

There is one catch though, and it is Alice herself, because Alice is unconfined_t. Her shell can read plaintext_t, because unconfined_t can read anything the Unix permissions allow, and it can write to the network, because unconfined_t can do anything, so rule 1, only the encryption program may turn plaintext into something that can be sent, is not true for programs Alice runs herself. It holds for the web server and for every other confined daemon, it does not hold for the user, and that is exactly the trade the targeted policy made. You can confine Alice, the reference policy has confined user domains, user_t and staff_t, which her login reaches by being mapped to the SELinux users user_u or staff_u, and in the strict policy every login landed in one of them (since Fedora 9 folded the two policies together you can do it for Alice alone), and if you have tried to run a desktop that way you know why the trade was made.

Where that left things

Where the Red Hat years left things comes down to three points.

  • By 2014 SELinux was on by default on a major distribution, it had a policy you could actually run, tools that explained it, a way to write your own rules, and two uses, confining network daemons and separating virtual machines and containers, where the labels did work nothing else could do.
  • Rule 2 is enforced on every Red Hat box out of the box. Rule 1 is enforced against every confined daemon and not against the user, and the user is the one everyone gave up on.
  • The setenforce 0 reflex never fully went away. It took a decade of tooling, and a website Major Hayden of Rackspace put up in 2013 telling administrators that every setenforce 0 makes Dan Walsh weep, to make turning it off stop being the easy option.

Every piece of policy above was written by a distribution for a general purpose operating system, on the assumption that there is an administrator on the other end who will read the denial, flip the boolean, run restorecon. Take away the administrator, put the operating system on a device with a fixed set of software, a billion copies, and a vendor who is the only one who will ever touch the policy, and every one of these tools is the wrong shape. Then the people who built SELinux were asked to put it on a phone.

Beyond servers

A phone has no administrator. It has a user who will never see an audit log and a vendor who ships the software once, with the policy baked in, to millions of devices that are all supposed to behave the same way, so the thing that made the targeted policy work, a human in the loop deciding what this particular machine needs, is simply not there, and the policy has to be right when the device leaves the factory because nobody is going to fix it afterwards.

Android is where the argument of this series got its largest test, mandatory access control on billions of general purpose computers running a policy nobody on the device can change, and it passed. It passed by doing the one thing the targeted policy had given up on, confining everything, and it could do that because three conditions held that never held on a server, the software is fixed, one organisation writes the core policy and binds everyone else who touches it with its neverallow rules, and the policy is checked by a compiler and a conformance suite rather than by an administrator.

SE for Android

Android had, by 2011, become the largest deployment of the Linux kernel there had ever been, and its security model was Unix permissions with a twist, every app runs as its own user, so the discretionary bits keep apps out of each other’s files. That is a reasonable idea and it has exactly the failure mode we started the series with, a kernel bug, a privilege escalation in one of the system daemons, and the attacker is root and the per app users are decoration. The rooting tools of that era were, one after another, exactly that.

Stephen Smalley and Robert Craig at the NSA put SELinux on it, made the first public release of SE Android in January 2012, and described it at NDSS in February 2013 as bringing flexible mandatory access control to Android. Most of the kernel side was already there, Android’s kernel was a Linux kernel and SELinux was in it, but not all of it, Binder, the IPC mechanism every Android process talks through, had no LSM hooks and needed new ones, and yaffs2, the flash filesystem of the time, needed its extended attribute support fixed and newly created files labelled automatically. The rest of the work was in userspace, labelling the filesystem images at build time (no restorecon on first boot for a device that must boot correctly the first time), teaching Android’s init to load the policy before anything else runs, labelling every app’s process and files, with a type chosen by what kind of app it is (third party, platform, system) and a set of MLS categories derived from its user ID, so that two apps of the same type still cannot touch each other’s files, and writing the policy itself for a system with a few dozen native daemons and a Java runtime that all the interesting code runs in.

Google picked it up, and Android 4.3, in the summer of 2013, shipped with SELinux in permissive mode, logging denials and enforcing none, to find out what breaks before anyone notices. (Samsung, whose Knox work was built on the same policy, actually shipped enforcing first, in a Galaxy S4 update that October.) 4.4, at the end of October 2013, switched to enforcing for four of the most exposed system daemons, the installer, the network daemon, the volume manager and the zygote that every app is forked from, and left everything else permissive, which is the move Fedora Core 3’s targeted policy had made nine years earlier, confine the few daemons that matter most and leave the rest alone. Android 5.0, released in November 2014, enforced it for everything, every process on the device runs in a domain, the only things left unconfined are the kernel itself and init, and the compatibility rules for the release require it. Confining everything is exactly what the targeted policy had given up on and what the strict policy had failed to do on the desktop, and Android has done it on every device since. It could do this because the software on the device is fixed, the vendor is the one writing the policy, and there is no administrator around to be inconvenienced by it.

neverallow

The tool that made a fixed policy possible at that scale is one that had been in the policy language for years, the neverallow rule. An allow rule adds a cell to the table. A neverallow rule says a cell must be empty, and the policy compiler refuses to build any policy in which some allow rule, anywhere, in any module, fills it. It is a constraint on the policy rather than on the system, and the reference policy has them too, but on a server they bind nobody, because the root who can load a module with a new allow rule is the same person who controls the compile.

On Android, neverallow is what the whole thing rests on. The core policy that Google ships is full of neverallow rules (no untrusted app may ever be made MLS trusted, the per app categories are what keep one app out of another’s data and a trusted app would step straight over them, no kernel module may ever be loaded from anywhere but the read only system and vendor images, no domain apart from the apps, the shell and the zygotes may ever execute a file that was not shipped in those images..etc.), and a device maker who wants to change the policy for their hardware, which all of them do, has to build their changes against those rules, and the build fails if they violate one. On top of that, the Compatibility Test Suite, which a device has to pass to ship with Google’s apps, checks the policy on the actual device against the same neverallow rules again, and this is how Google, who doesn’t build most of the phones, got every one of them to enforce the same minimum. This is the first time in this series that the policy is a contract between two organisations rather than a file on one machine, and the neverallow rules are the clauses.

Android's policy: Google's platform policy and the device maker's vendor policy are compiled together on the device, neverallow rules are checked by the compiler and again by the compatibility test suite, and every app gets a domain plus categories platform policy, written by Google plat_sepolicy.cil every system daemon, every app domain, the neverallow rules, plaintext_t exports attributes for vendors to use vendor policy, written by the device maker vendor_sepolicy.cil its own daemons, the encrypt service, its hardware, using only those attributes, never the platform's private types compiled together, on the device, into one binary policy the kernel loads at boot the compiler refuses the whole thing if any allow rule, from either side, violates a neverallow neverallow netdomain plaintext_t:file read; a vendor allow rule for it is a build error Compatibility Test Suite re-checks the same rules on the shipped device untrusted_app:s0:c512,c768 every app, categories per app hal_encrypt_t the encryption service no unconfined domain at all except the kernel and init
Android’s policy since Treble. Google’s half and the device maker’s half are compiled together on the device, the compiler and the compatibility suite both enforce the neverallow rules, and every app runs in a domain with its own categories.

Treble, and the split

That contract had been in writing since Android 5.0, whose compatibility definition forbids a device maker to modify, omit or replace any of the neverallow rules in Google’s policy, and Android 8, in 2017, gave it an interface, under the name Project Treble, which is remembered as the thing that let vendors update Android without redoing their hardware work, and which was, for the policy, a split. The platform policy, Google’s, and the vendor policy, the device maker’s, became two separate files, built separately, and combined on the device. The vendor’s policy may only refer to the platform’s types through a published set of attributes, a stable interface exactly like the reference policy’s interfaces, and the platform’s private types are, to the vendor, invisible. A vendor can add a domain for their camera daemon and give it access to their camera device node, and cannot give it access to the platform’s keystore, because the type is not in the interface.

To make that mechanically possible the two halves are written, and combined, in a language designed for exactly this kind of programmatic manipulation. CIL, the common intermediate language, was built at Tresys in the early 2010s and released as part of the SELinux userspace in 2015, and it is what the policy language SELinux has used since 2000 compiles into before the kernel sees it. It is uglier to read than the old language (it looks like Lisp) and far easier to generate, transform and merge, which is why semodule uses it underneath for every module on a modern Fedora, and why Android’s build system could take a platform policy and a vendor policy written against different versions of the interface and produce one binary policy for the kernel. When you see plat_sepolicy.cil and vendor_sepolicy.cil on a device, that is Treble’s contract, in the language that made it enforceable.

Embedded Linux

Phones are the loud case. The quiet one is everything else that runs Linux without an administrator, routers, cars, industrial controllers, medical devices, and here the story is less finished. Much of that world builds its Linux with Yocto, and Yocto has a layer, meta-selinux, that carries the SELinux userspace, the kernel configuration, a choice of reference policy flavours, and the build time labelling of the image, so that a device can leave the factory enforcing a policy the way an Android phone does. What it cannot carry is the policy for your device, because there is no Google writing a platform policy for industrial controllers, and every product team that turns it on finds itself, in miniature, doing what Red Hat did in 2004 and Google did in 2013, working out which of their daemons are worth confining and writing the rules, with the difference that they will never get to flip a boolean after shipping. According to my current understanding, a lot of these projects get as far as running in permissive mode and never make the switch to enforcing, which would put a good part of embedded Linux in 2026 roughly where Android was with 4.3.

Running the example, on a device

The web server is now a service exposed to the network, an app or a daemon, in its own domain. Alice’s plaintext is a file the device has to protect, with its own type, plaintext_t, declared in the platform policy so that every device carries it. The encryption program is a service in the vendor half, in its own domain, and the sending program is the network daemon. The allow rules are the same as they have been since part 1, encrypt may read plaintext and write ciphertext, send may read ciphertext, the web server’s domain has neither. And then one more line, in the platform policy, neverallow netdomain plaintext_t:file read, no domain that can open a network socket may ever read the plaintext. Now a vendor who, three years and two teams later, adds an allow rule for their diagnostic tool that happens to give the web daemon that access, does not ship a vulnerable device, they get a build error, and if they somehow get past that they fail the compatibility suite. The policy is not just enforced by the kernel on this device, it is enforced by the compiler on every device, and there is no unconfined_t for Alice’s own programs to fall back into because there is no Alice, there is a user who taps things and a fixed set of programs that all have domains.

Where that left things

Android’s result comes down to three points.

  • Android enforces both rules, on every device, against every process, which no server ever did. A great deal of the exploit chains of the last decade find their vulnerable daemon and then find that its domain cannot do the next thing the chain needs, and the neverallow rules make it much harder for a device maker’s policy to quietly hand it that permission.
  • It could only do that because the policy is a contract, its core written by one organisation, its limits imposed on every device maker, checked by a compiler and a conformance suite, and never touched by whoever holds the device.
  • Embedded Linux has the same tools and none of the contract.

That is one way to get a policy, labelled, centrally written, and held in place by a build system. It is not the only one, and it is not the one most of the rest of the Linux world chose. Look at Ubuntu, or a Mac, and the confinement is there but the labels are not, the policy is written in terms of file paths, and increasingly it is not written by a distribution at all but by the program’s own author, or by the program itself, at runtime, for itself. Crispin Cowan, who started the Linux Security Modules project in 2001 and co-wrote the paper that described its hooks, had been building that other approach since before SELinux was merged, and the argument between the two, labels against paths and who gets to write the policy, is still going.

The other LSMs

Android is a real success, and most of the rest of the Linux world looked at it and chose something else. Ubuntu does not run SELinux, Debian does not by default, SUSE did not until 2025, and the sandbox on every Mac and iPhone looks nothing like it. The reason is not that those people failed to understand SELinux. It is that SELinux answered two questions one way, whether a policy should be about labels or about names, and who should write it, and there are respectable answers on the other side of both. Where the answers come apart is our example.

The hooks

The reason all of these can coexist at all is the decision Linus Torvalds made in 2001, told in part 2, to ask for hooks instead of a security model. The Linux Security Modules framework that came back is a list of places in the kernel, a few hundred of them now, where the kernel stops just before doing something to an object and asks whoever is loaded behind the hook whether it may. The hooks run after the ordinary Unix permission check, not instead of it, so a module can take access away and, root’s capability checks aside, never add it, and SELinux was only the first big module behind them. Crispin Cowan, who started the project, is also the lead author of the mechanism that has spent twenty years as SELinux’s main rival, which tells you that the hooks were never meant to have one answer.

A system call passes the Unix permission check, then a Linux Security Modules hook, and whichever module is loaded answers the hook by its own rule the web server calls open("plaintext") Unix permission check owner, group, mode, root passed LSM hook security_file_open() whichever module is loaded answers one major module at a time, plus any number of small ones SELinux, Smack by the object's label AppArmor, TOMOYO by the object's path Landlock by the process's own rules BPF LSM by code you attach the hook runs after the Unix check, so a module can only take access away, never add it Yama and the other small modules run alongside the major one Every mechanism in this part sees exactly the same call from the web server, and each decides it by a different rule. The hooks are what Linus asked for in 2001 instead of one built in model.
The Linux Security Modules hooks. The web server’s open call passes the Unix check, reaches the hook, and whichever module is loaded decides by its own rule. Every mechanism below sees the same call.

So every Linux mechanism below is a different module answering the same hooks, which means every one of them sees exactly the same request our example has been generating since part 1, the web server asking to open Alice’s plaintext, and decides it by a different rule.

AppArmor

Cowan’s company was WireX, in Portland, later renamed Immunix after its product, and in the late nineties it shipped a hardened Red Hat built around the StackGuard compiler, with a thing called SubDomain, described at LISA in 2000, as its access control. SubDomain confined a program, not a user and not a label, a program, identified by its path, to a list of the files it was allowed to touch, also identified by their paths, each with a mode (later versions added capabilities and network access to the list). The web server’s profile in the 2000 paper says the web server may read its document root, its own configuration and the libraries and system files it loads, may write its logs, and that is all. No labels on disk, no relabelling, nothing to do to a file for it to be protected, you just don’t put its path in the profile. The pitch, then and now, was that an administrator can read a profile and know what it means, which is not something anyone has ever said about the reference policy.

Novell bought Immunix in May 2005, renamed SubDomain to AppArmor, released it as open source at the start of 2006, and shipped it on by default in SUSE Linux 10.1 that May. Novell then let the team go in 2007, Ubuntu has shipped it on by default since 7.10 that same October, Canonical took over the maintenance in 2009, and it went into the mainline kernel in 2.6.36 in October 2010, seven years after SELinux. If you run Ubuntu or Debian today, or a SUSE machine installed before 2025 (openSUSE switched new installations to SELinux that February), this is the module answering the hooks on your machine, and the profiles for the daemons that ship with the distribution come from the distribution, exactly as the targeted policy does, with a program that has no profile running unconfined, much as unconfined_t does.

Labels against paths

The argument between the two camps happened in public, on the kernel mailing list, from April 2006, when AppArmor was first submitted, until well into 2007, it was not polite, and both sides were right about the thing they were arguing about.

The SELinux side’s objection is about what a name is. A path is not a property of a file, it is a property of a directory entry, and a file can have several (hard links) or none (an open file that has been unlinked), and can change its name without changing at all (rename). A policy that says the web server may not read /home/alice/secret/plaintext is a policy about a string, and the moment there is a second string for the same bytes the policy has nothing to say. That is not a corner case, it is how Unix works, and a reference monitor that is consulted on every access but decides by a name that ln can change is enforcing a policy about the wrong thing. There were also uglier arguments about how a module even finds out the path of an object deep in the kernel, which is what most of the mailing list traffic was actually about.

The AppArmor side’s reply is about what a policy is for. A policy nobody can write is a policy nobody enforces, and what people who administer machines think in is paths, not types, and a mechanism that lets a developer ship a profile with their package that any of their users can read and audit is worth a hard link or two, and anyway you can confine the program that would create the hard link. Torvalds, characteristically, said that the argument was pointless and that the kernel should carry both, and it does.

One file with two names, one under Alice's home and one under the web root, and how a path based profile and a label based policy each decide the web server's read /home/alice/secret/plaintext the name Alice uses /var/www/html/notes.txt a hard link made by an unconfined shell one inode, the same bytes security.selinux = plaintext_t web server wants to read a path based profile /var/www/html/** r, the string matches, so the read is allowed a label based policy httpd_t has no rule for plaintext_t the label came with the inode, so the read is denied
One file, two names. A path based profile decides by the string and allows the read through the second name. A label based policy decides by the label on the inode and denies it through either.

Running the example

An AppArmor profile for the web server that does not mention Alice’s directory keeps the web server out of it, and it does so without anyone labelling a single file, which is a real advantage on a machine with a hundred thousand of them, and the web server cannot make itself a hard link to get around it, because creating links is one of the things a profile mediates. Then something that can write to the web root and has no profile, Alice’s shell if she looks after the site, or a root shell, makes a hard link to the plaintext inside /var/www, which works whenever /home and /var/www are on the same filesystem, as they are on a machine with one big partition (a plain mv does the same thing and leaves no old name behind), and AppArmor’s stated design is that it trusts anything without a profile, so the link gets made, and now the web server can read the plaintext, because the profile allows /var/www and the plaintext is in /var/www, as far as the string is concerned. SELinux would deny it, because the link points at an inode labelled plaintext_t and the label came along. Cowan’s own answer, on the mailing list in 2006, was that this is by design, if the policy lets you read a path you get to read it even if it is a hard link to something else, and the unconfined process that made the link was trusted by definition. Which of those you find more alarming depends on whether you think the attacker is outside the web server or already on the box, and reasonable people have been disagreeing about that for twenty years.

The smaller ones

Three more modules went into the kernel around the same time, and each is a different answer to the same two questions.

Smack, Casey Schaufler’s Simplified Mandatory Access Control Kernel, merged in 2.6.25 in 2008, is labels with the type enforcement machinery stripped down to the bone. A label is a short string, the rules are triples, this label may do this to that label, and there are no roles, no users, no transitions to speak of, and no policy language, because the point was that a policy for a device could fit on a page. Samsung and Intel used it for Tizen, and it is in Automotive Grade Linux, which is to say it went where SELinux was considered too big.

TOMOYO, from NTT Data in Japan, merged in 2.6.30 in 2009, is path based like AppArmor, with a different idea about domains, a process’s domain is the chain of programs that led to it (init, then the shell, then the web server), so that the same program invoked two different ways can have two different policies, and with a learning mode that watches what a program does and writes the policy for you, which is audit2allow turned into a way of life.

Yama, from Kees Cook, merged in 3.4 in 2012, isn’t an access control model at all, it is a small set of restrictions on a couple of dangerous operations, most famously limiting which processes may ptrace which others, and it is here because it is the first module that was explicitly designed to run alongside another one, which is the thing the framework had not originally allowed. Modules that own the labels on objects, the big ones, could only be loaded one at a time, the kernel’s own documentation still says a boot gets any number of minor modules and at most one major one, and getting the framework to stack the big ones properly, so that a machine could run SELinux and something else together, has taken Schaufler more than a decade and was still being argued about with Torvalds in 2024.

The program writes its own policy

The last two are the interesting ones, because each answers the second question differently from everything above, Landlock by handing the policy to the program, the BPF LSM by turning the policy into code.

Every mechanism so far has the policy written by someone other than the program’s author, the distribution, the device maker, the administrator, and applied from outside. Landlock, by Mickaël Salaün, merged in 5.13 in 2021, lets the program do it to itself. A process builds a ruleset, read access under this directory and write access under that one and nothing else on the filesystem, applies it, and from that instant it, and every child it ever creates, is confined to that ruleset, with no way to remove it, and no privilege required to set it up. The object of confinement is the process, not a label and not a path string (Landlock rules refer to directories the process has already opened and the check follows the hierarchy of the path being used, so a link somebody else makes inside an allowed directory is readable, just as under AppArmor, and what Landlock stops is the sandboxed process itself linking or renaming files across hierarchies), and the author of the policy is the author of the program, who is the person who actually knows what the program needs. OpenBSD had gone the same way a few years earlier with pledge and unveil, and the macOS sandbox below is the same idea with a profile file.

The BPF LSM, by KP Singh, merged in 5.7 in 2020, drops the idea of a policy language altogether. You attach a small program, written in eBPF, to any of the security hooks, and the program decides, in code, with access to whatever kernel state it likes. It is policy flexibility in the most literal sense the Flask paper could have meant, the security server is whatever you compile, and it is being used, at the companies that run very large fleets, to enforce rules that no static policy language could express, and it is seen, by the people who worry about such things, as the thing that finally makes the hooks into an API for both defenders and rootkits.

Outside Linux

The same argument ran in parallel on the BSDs. Robert Watson’s TrustedBSD MAC framework went into FreeBSD 5.0 in 2003, a few months before SELinux went into 2.6, and it is the same napkin as DTOS, a framework of hooks in the kernel and pluggable policy modules behind them, Biba, MLS, and a port of the Flask security server itself as one of the options. Apple took the TrustedBSD framework into Mac OS X and built the sandbox on it in 2007, the thing that was called Seatbelt internally, where a program is confined by a profile written in a small Scheme-like language, mostly in terms of paths, and applied either by the system or by the program calling sandbox_init on itself. Every app from the Mac App Store has run under it since 2012, every app on an iPhone runs under a descendant of it, and the profiles are written by Apple and by the app’s developer, never by the user. Which is to say the most widely deployed sandbox outside Android answered the first question with names and the second with the developer, and did it on a kernel whose hooks came from the same family as Linux’s.

Where that left things

The argument between the modules comes down to three points.

  • There are two questions, whether the policy talks about labels or about names, and whether it comes from the platform, from the developer, or from the program itself at runtime. SELinux answered labels and the platform, and it won in the places where there actually is a platform to write the policy, Red Hat’s servers and Android. AppArmor and the macOS sandbox answered names and the developer, and they won in the places where the person who ships the software is also the person who knows what it needs. Landlock and pledge let the program write the rules for itself, and they are the youngest of the bunch.
  • Against our example, labels with a central policy can say the whole sentence, the web server may not, and nobody but the encryption program may. Names with a developer’s profile can say the first half well and the second half only if every program on the box has a profile, which none of them does. A program sandboxing itself can say only, I will not, which is a promise the encryption program can make about itself and no mechanism can make the web server keep.
  • Each of them is a reference monitor, each is consulted on every access at the same kind of hooks, and each enforces a policy that a different person wrote.

What forty years bought

This series has brought us from a kernel that could not even ask the question to three ways of answering it, and the question now is what that was worth. The answer is that forty years bought exactly one of the two things the Inevitability of Failure paper asked for, and both the success and the failure were visible in 1998 to anyone who read it carefully.

What survived

The test of an idea is whether it is in the kernel you are running right now without you having had to think about it, and by that test five ideas from this series survived.

  1. Policy separated from mechanism survived completely. It is the DTOS napkin from part 2, it is the reason a Fedora machine and an Android phone and a hardened container host all run the same security server code with different rules loaded into it, and it is the reason the hooks could carry AppArmor and Smack and Landlock as well. Nobody builds an access control mechanism with the policy inside it any more, and the one part of Unix that still does, the permission bits, is the part everyone in this series was trying to get past.
  2. Labels survived, on Linux at least. Every file on a SELinux system has a type in an extended attribute, every process has a context, and the answer to whether the label or the name should be the thing you write policy about turned out to be that the label is the only thing that is actually a property of the object, which is why sVirt and container separation work and why the hard link argument between the modules never went away. Paths won the usability argument and labels won the correctness one, and the kernel carries both.
  3. The access vector cache survived so thoroughly that it stopped being an idea, it is just how a reference monitor is built now, SELinux’s cache is the one every fast check is modelled on, and the sequence number protocol from Flask is why you can load a new policy on a running system and have it apply to files that were already open.
  4. Type enforcement survived as the workhorse. Not levels, not roles, not the fancier things the policy language can express, the two tables from 1985, a domain, a type, an allow rule, and a transition. Almost every line of policy that does real work on a real machine is one of those, and Boebert and Kain’s assured pipeline, the plaintext to ciphertext example we have run in every part, is a thing you can type into a Fedora machine in twenty lines and have the kernel enforce.
  5. neverallow survived in a way nobody planned, as a contract. On Android it turned a policy from a file on one machine into an agreement between a platform and a thousand device makers, enforced by a compiler, which is a thing the Orange Book’s authors would have recognised as an evaluation and would not have believed could be automated.

What did not

Four ideas did not survive as designed, and each of them died for the same reason, the people it was designed for were not the people who ended up running it.

  1. Confining the user did not survive, at least not the way the strict policy meant it. The strict policy itself was folded into targeted in Fedora 9, and the confined user domains, user_t and staff_t and the rest, are still in the policy and still documented, but every distribution logs its users in unconfined by default and almost nobody changes that, because a desktop is a machine whose whole purpose is to let one person run arbitrary programs on their own data, and a policy that gets in the way of that gets turned off. Android confines every app and gets away with it because an app is not the user’s program, it is a vendor’s, delivered through a store, with a manifest, and the user never sees a denial. The second half of our sentence, nobody but the encryption program, is enforced today on servers and phones and not on the laptop you are reading this on, and I don’t think that is going to change.
  2. Multilevel security did not survive outside the buildings it was designed for. It is in RHEL, it is evaluated, it is used where the policy really is the military’s policy, and everywhere else the sensitivity is stuck at s0 and the level field has been put to a different use, the random pair of categories that keeps one virtual machine or container away from the next and the categories that keep one Android app away from another. The lattice was the wrong abstraction for everyone who didn’t already have clearances, which is close to what Boebert and Kain said about hierarchical integrity in 1985, and it took the rest of the field about twenty-five years to agree.
  3. The microkernel did not survive, twice. DTOS and Flask were built on Mach and Fluke because a microkernel’s message passing was the natural place for a reference monitor, and the mechanism that actually shipped was a set of function calls scattered through a monolithic kernel, because that was the kernel people ran. The small verifiable security kernel of Anderson’s report was built more than once, SCOMP and GEMSOS were evaluated at A1, and it got its first machine checked proof down to the code as seL4 in 2009, which is a genuine achievement, and it runs almost nowhere that our web server runs. The reference monitor that mediates most of the world’s file accesses is a few hundred hooks in Linux, and it is tamperproof and always invoked only to the extent that the other thirty million lines of kernel are correct, which is exactly the compromise Anderson said not to make. It was made anyway, because the alternative was an architecture nobody used.
  4. Policy flexibility, the headline claim of the Flask paper, survived in a narrower form than promised. The security server in Linux enforces one family of policies, type enforcement with roles and optional levels, and it has since 2000. The genuinely different policies, the ones the generalized security policy specification catalogued, ended up as different modules, or, lately, as eBPF programs, which is flexibility of a kind, but not the kind where you swap the policy under a running system and the object managers don’t notice.

The record

Did it work, in the plain sense, did it stop attacks? The honest answer is yes, in specific documented cases, and also that the cases where it didn’t are the same shape every time.

The ones where it did tend to be container escapes and Android privilege escalations, which is to say the two places where the policy was written by a platform and left on. When the runc container escape of early 2019 was disclosed, Red Hat’s advisory said that SELinux in enforcing mode completely prevents the exploit on its systems, because the container’s domain has no permission to overwrite the runtime’s binary, which is our web server unable to read the plaintext with different nouns, and Dan Walsh had written the same thing about a Docker escape two years earlier, that SELinux was the only thing standing between a process inside the container and the host’s filesystem. On Android, Smalley and Craig’s 2013 paper walked through seven published rooting exploits and showed the policy stopping or containing every one of them (psneuter still got its root shell, it just got it in a confined domain), Google’s own timeline slides repeat the figure as seven exploits prevented or mitigated, and at the Linux Security Summit in 2018 two of Android’s kernel security engineers, Jeff Vander Stoep and Sami Tolvanen, put up a slide quoting Mark Brand of Google’s Project Zero, who said that by far the most effective mitigation work on Android in the previous three years had been attack surface reduction, and that the tightening of the SELinux policy and the addition of seccomp both left an attacker needing more bugs in a smaller surface. The chain finds a vulnerable daemon and then finds that the daemon’s domain cannot do the next thing the chain needs.

There is also one case where SELinux itself was the hole, and it is the honest counterexample. In 2009 Brad Spengler’s exploits showed that on a kernel with SELinux enabled, the kernel’s own protection against null pointer bugs, the rule that ordinary processes may not map the lowest pages of memory, had been handed over to the SELinux policy to decide, and Red Hat’s policy, for backwards compatibility, let the unconfined domain map them anyway, so that a class of kernel bugs was exploitable for root on machines with SELinux switched on and harmless on machines with it switched off. Red Hat fixed the policy that September and the kernel that November (the kernel side is CVE-2009-2695, and the upstream fix makes sure a security module can no longer be more permissive than the kernel’s own check), and the lesson stuck, a policy is code, and code has bugs, and a mandatory policy with a bug in it is mandatory in both directions.

Where the trust sits: a confined program's compromise stops at the policy, a kernel bug goes under the hooks and edits the monitor, and the newer designs rebuild the security kernel one layer below Linux the program, confined by a policy it did not write the web server in httpd_t, or under a profile the policy, or the sandbox a program applies to itself SELinux, AppArmor, Landlock, seccomp, pledge the reference monitor, a few hundred hooks always invoked, tamperproof only if the kernel is correct the Linux kernel, thirty million lines the thing every mechanism above assumes is correct one layer down, Anderson's security kernel again TrustZone, enclaves, confidential VMs, pKVM under Android a bug in the web server stops at the policy, the domain cannot do the next thing a bug in the kernel goes under the hooks and edits the monitor, setenforce 0 from inside the answer put what you care about where the kernel cannot reach it Everything in this series lives in the top three layers and rests on the fourth. The fifth is the same small, tamperproof, always invoked kernel from 1972, built again because the first one grew too big, and it will need a policy too.
Where the trust sits. A compromised program stops at the policy. A kernel bug goes under the hooks and edits the monitor. The newer designs rebuild the small security kernel one layer below Linux.

The ones where it didn’t are almost always below the hooks. A kernel memory corruption bug, the sort that lets an attacker write to arbitrary kernel memory, does not go through the reference monitor, it edits it, and the usual finishing move of an Android kernel exploit is to switch the policy off or rewrite it, a 2022 research exploit from GitHub’s Security Lab, one bug in a Qualcomm GPU driver on a Samsung phone, does the first, it writes a zero over the flag that says enforcing, and an in the wild Samsung chain that Google’s Project Zero took apart the same year does the second, it overwrites the policy loaded in kernel memory so that its own process gets the context of the volume manager. Dirty COW in 2016 went under the hooks a different way, a race in copy on write that let a process write to files it could only read without a write check ever being made, which is why the Linux Foundation said at the time that it was not the kind of bug SELinux or AppArmor could do much about. Nothing in this series helps with that, because everything in this series assumes the kernel is the thing you trust, which is the assumption Anderson was uneasy about in 1972 and the one the whole design rests on. The other failure is the human one, the machine where the first response to a denial was setenforce 0 and nobody set it back, and there is no mechanism for that, only the decade of tooling from the Red Hat years that made the denial explicable enough that turning it off stopped being the easy option.

So the verdict on the record is narrower than either the advocates or the sceptics like. Mandatory access control in the kernel reliably contains the compromise of a confined program, on the systems where someone wrote and kept the policy, it can keep some kernel bugs out of reach by keeping confined programs away from the code they live in, and once the kernel itself is compromised it does nothing, and it does nothing on the systems where it was switched off. Both of those limits were known in 1998, and both are what the newer designs work around.

Where it is going

Two things are happening now, and they are the two limits above being worked around from different directions.

The first is that the policy is moving into the program. Landlock, seccomp, pledge, the macOS sandbox applied from inside, the eBPF programs that a service loads to describe its own behaviour, all of these are a program telling the kernel what it will never do, at startup, before it has read any untrusted input, and in new software, more and more, the developer writes the policy, because the developer knows what the program needs and the distribution does not. This gives up the second half of our sentence, a self imposed sandbox cannot say nobody but me, and it gains the thing the strict policy never had, a policy that is correct for the program because the program’s author wrote it. My guess is that the labelled, platform written policy stays where it is, on the platforms, and everything else goes this way.

The second is that the trust is moving out of the kernel. If the reference monitor’s weakness is that a kernel bug edits it, then the answer that the hardware people arrived at is to put the thing you care about somewhere the kernel cannot reach, a separate world on the same chip (TrustZone, on every phone), an enclave the kernel schedules but cannot read (SGX and its successors), a virtual machine whose memory the hypervisor itself cannot see (the confidential computing designs from AMD and Intel), or on Android a small protected hypervisor underneath the Linux kernel so that a kernel compromise no longer reaches what runs in the protected virtual machines. Every one of those is Anderson’s security kernel, small, tamperproof, always invoked, built again one layer down, because the layer it was built in the first time turned out to be thirty million lines long. And every one of those will, in time, need a policy, and someone will have to decide whether it is written in labels or names, and by whom, and the series will start again.

Closing Points and Remarks

Several points are worth noting though

  • The question that could not even be asked back in part 1, may this program (as opposed to this user) read this data, is now a question every Linux kernel can answer on every single access, from a policy that the kernel doesn’t contain and that root can’t override wherever root is itself confined, which on Android is everywhere, and the machinery for asking it, hooks in the kernel and a policy kept outside the mechanism, is now in every serious operating system. That is a genuine change in what a computer is, even if most people using one will never notice it.
  • What it did not buy is the second thing the Inevitability of Failure paper wanted, mandatory security that everybody actually runs. It runs where a platform writes the policy, and where the platform is a person, it doesn’t, and the field has largely stopped pretending otherwise and moved on to letting programs confine themselves. Whether that is a retreat or the correct answer arrived at late, I honestly do not know.
  • The encryption pipeline we have carried since part 1 is enforceable today, by the kernel, on an ordinary machine, in twenty lines that a mere mortal can read, while back in 1985 the only place you could type that sentence into was a coprocessor in Minnesota, and for me at least that alone makes the forty years worth it.

Disclosure

Nothing in this series comes from privileged information, it is all based on the published papers and reports in the reference section.

Please note that this series is my own reading of the history; I might have skipped or been extremely loose about how specific systems worked. If you do find incorrect information, please contact me so I can correct it.

References