May 20, 2016

grsec in Docker - first practical experience

Intro

So what is grsecurity?

Grsecurity® is an extensive security enhancement to the Linux kernel that defends against a wide range of security threats through intelligent access control, memory corruption-based exploit prevention, and a host of other system hardening that generally require no configuration. It has been actively developed and maintained for the past 15 years. Commercial support for grsecurity is available through Open Source Security, Inc.

First practical experience

Below you can find the problems & solutions (or workarounds) which I experienced while running the grsec enabled kernel.
Please note that I am running most of the apps in Docker.
You can find my dockerized apps here, some of which already are grsec-friendly.

webcam does not work

I have passed the webcam - /dev/video0 device, but Firefox, Chrome, cheese or mplayer cannot stream the video from the webcam.

X11 error: BadAccess (attempt to access private resource denied)
X11 error: BadShmSeg (invalid shared segment parameter)

To remedy the situation /dev/shm should be mounted as a RW volume. The best way to achieve this is docker run --shm-size=128m ...

apt-get, yum, keepass2 (mono) are terribly slow

With the grsecurity patched kernel, you may experience notably poor performance when using apt-get, yum or keepass2 (mono). The workaround was to lower the “open files” limits as follows

ulimit -n 1024

or better

docker run --ulimit nofile=1024:2048 yourapp

Alternatively, you could pass --default-ulimit nofile=1024:2048 to your docker daemon, that way all containers would have lowered user limits for open files. Which would be also helpful when running docker build.

grsec: denied RWX mprotect of by /path/to/someapp

To remedy this, you will need to disable MPROTECT. There are two ways of doing so, first one is by using the filesystem labels (xattrs) and the other one is by converting the binary headers PT_GNU_STACK into PT_PAX_FLAGS.
When using the first one, please remember to preserve the labels when doing backups (e.g. see tar --xattrs).

With paxctl-ng (via xattrs)

paxctl-ng -lmv /path/to/someapp

With setfattr (via xattrs)

setfattr -n user.pax.flags -v "m" /path/to/someapp

With paxctl (converting the binary headers)

paxctl -c -v -m /path/to/someapp

I suggest using paxctld service where you could configure what you need via /etc/paxctld.conf file:

/usr/local/bin/docker-compose               m

/usr/bin/python2.7                          E

/usr/bin/grub-script-check                  E
/usr/sbin/grub-mkdevicemap                  E
/usr/sbin/grub-probe                        E

/usr/bin/qemu-system-i386                   m
/usr/bin/qemu-system-x86_64                 m

/usr/bin/pulseaudio                         m
/usr/bin/compiz                             m
/usr/bin/cheese                             m

/usr/lib/libreoffice/program/soffice.bin    m

/usr/lib/firefox/firefox                    m
/usr/lib/firefox/plugin-container           m
/opt/google/chrome/chrome                   m
/usr/bin/youtube-dl                         m

/usr/bin/mono-sgen                          m

/usr/share/atom/atom                        m
/usr/share/atom/resources/app/apm/bin/node  m

/opt/HipChat4/lib/HipChat.bin               m
/opt/HipChat4/lib/QtWebEngineProcess.bin    m

I am wondering whether there a way to keep the MPROTECT. Please let me know in the comments or email me.

grsec and the caps

CAP_SYS_ADMIN is required to run the Google Chrome.
However, it is NOT required if you run chrome with --no-sandbox flag which is highly discouraged if not running in a unprivileged container.
Read more on CAP_SYS_ADMIN

For more application specific settings, refer to the official doc

ApplyLayer exit status 1 stdout: stderr: chmod /bin/mount: permission denied

followed by
grsec: denied chmod +s of /var/lib/docker/overlay/de1.../root/bin/mount by /usr/bin/docker

or
failed to register layer: ApplyLayer exit status 1 stdout: stderr: operation not permitted
followed by
grsec: denied mknod of /var/lib/docker/overlay/c27.../root/dev/agpgart from chroot by /usr/bin/docker

or a pretty much useless error
ApplyLayer exit status 1 stdout: stderr: operation not permitted
followed by
grsec: use of CAP_SETFCAP in chroot denied for /usr/bin/docker

To build the Docker image, I had to disable the following grsec protections.
Modify the /etc/sysctl.d/grsec.conf as follows:

kernel.grsecurity.chroot_deny_chmod = 0
kernel.grsecurity.chroot_deny_mknod = 0
kernel.grsecurity.chroot_caps = 0        # related to a systemd package/CAP_SETFCAP

grsec: denied resource overstep by requesting 28 for RLIMIT_NICE against limit 0

This is just an informational message, which means that the denial is not caused by grsecurity. A vanilla kernel would have denied it as well – grsecurity is only logging the event. If you don’t want to see those kinds of logs, then disable the resource logging feature you enabled. (credits to Brad aka Spender)

So basically this message means that your application is requesting more resources than is allowed by the limits set in your system. These limits are set in /etc/security/limits.conf or with ulimit command.

If you still want to get rid of that message and give your app the scheduling priority it wants, then run your container this way:

docker run --ulimit nice=28 yourapp

grsec: denied kernel module auto-load of xfrm4_mode_tunnel by uid 100

Having the IPSec in a Docker container while running a kernel with enabled grsec can bring some troubles.
To solve them, simply load the necessary IPSec modules on a host server boot:
for i in xfrm-mode-2-1 xfrm-mode-10-1 xfrm-type-2-50; do /sbin/modprobe -v $i; done

grsec: From [IP redacted]: denied untrusted exec (due to not being in trusted group and file in group-writable directory) of /path/to/someapp

This means that the directory is group-writable where your app is trying to run.
Usually, it is enough to remove group-writable permissions chmod go-w /path/to

grsec: denied untrusted exec (due to not being in trusted group and file in non-root-owned directory) of /path/to/someapp

This means that the /path/to supposed to be owned by the root.
It is a good idea to review your app’s deployment and move your app to another directory and owned by the root user.

Or you could add your user to the trusted group:

# grep TPE_GID /boot/config-4.6.4-grsec 
CONFIG_GRKERNSEC_TPE_GID=64040

# sysctl kernel.grsecurity.tpe_gid
kernel.grsecurity.tpe_gid = 64040

groupadd -r -g 64040 grsec-tpe
usermod -aG grsec-tpe youruser

RBAC system

I suggest you to run the gradm2 -C on your system in order to see whether you have it enabled.
Refer to the official Grsecurity documentation to learn more about the grsec / PaX / RBAC.

Final words

I am impressed by the grsec’s capabilities and wondering, why it is not fundamentally adopted by the major Linux distro makers like the RedHat, Ubuntu, openSUSE, …?

Here is an interesting paper Grsecurity Developer Spender’s Feelings on the State of Linux Security

Links

I would like to add a set of MUST READ links here