February 02, 2017

ZFS on LUKS in Ubuntu 16

As a source material I was referring to Ubuntu 16.04 Root on ZFS.
Though it did not explain how to use LUKS with ZFS.

Below are the necessary steps to install ZFS on LUKS as your primary filesystem in Ubuntu 16.04.1 LTS (64-bit).

I have used QEMU/KVM so that you can easily test this by yourself, and when confident, apply it on a real hardware.

Start Ubuntu Live

Create a fake disk

qemu-img create -f qcow2 fake.disk 4G

Start Ubuntu

qemu-system-x86_64 -enable-kvm -m 1024 -drive file=fake.disk,if=virtio,serial=VIRTDISK1 \
	 -cdrom ~/Downloads/ubuntu-16.04.1-desktop-amd64.iso \
	 -net nic,model=virtio -net user,hostfwd=tcp::2222-:22

To boot from CDROM again use -boot once=d
Press Ctrl+Alt+2 (to open qemu monitor/console), where you can use system_reset command which can be handy.

Once Ubuntu 16 is started, choose “Try Ubuntu”.

Now you can either continue working in the same terminal, or work remotely.

sudo -i
dpkg-reconfigure tzdata
passwd ubuntu
apt update && apt -y install openssh-server

ssh -p2222 ubuntu@localhost
sudo -i

Install basic tools

apt -y install debootstrap gdisk zfsutils-linux

Prepare your disk

Running the following commands will cause your disk be GPT-formatted.
The first partition is going to be your primary, LUKS encrypted and the largest one.
The 2nd partition will be used by a GRUB installer for writting the secondary bootloader (stage2 GRUB loader) used in non-UEFI mode.
3rd partition is an EFI System Partition (ESP), is to make system bootable in UEFI mode (see “UEFI booting” below).
4th partition will be used by GRUB to store its configuration - /boot/grub/grub.cfg file and the GRUB modules to support multiple boot options.
The 9th partition is Solaris Reserved 1. It is required by ZFS (yet, I don’t know why).

wipefs -a /dev/disk/by-id/virtio-VIRTDISK1
sgdisk -a1 -n2:34:2047 -t2:EF02 /dev/disk/by-id/virtio-VIRTDISK1
sgdisk -a1 -n3:1M:+512M -t3:EF00 /dev/disk/by-id/virtio-VIRTDISK1
sgdisk -a1 -n4:+512M:+512M -t4:8300 /dev/disk/by-id/virtio-VIRTDISK1
sgdisk -a1 -n9:-8M:0 -t9:BF07 /dev/disk/by-id/virtio-VIRTDISK1
sgdisk -a1 -n1:+1G:-256M -t1:8300 /dev/disk/by-id/virtio-VIRTDISK1

sgdisk -p /dev/disk/by-id/virtio-VIRTDISK1 will print similar output to

Number  Start (sector)    End (sector)  Size       Code  Name
   1         5244928         7847901   1.2 GiB     8300  
   2              34            2047   1007.0 KiB  EF02  
   3            2048         1050623   512.0 MiB   EF00  
   4         2099200         3147775   512.0 MiB   8300  
   9         8372190         8388574   8.0 MiB     BF07

Partitions:

1: 8300 - Linux filesystem, to be LUKS encrypted and then used as primary ZFS pool
2: EF02 - BIOS Boot Partition, to store secondary boot loader (stage2 grub loader) (for non-UEFI)
3: EF00 - EFI System Partition (ESP), to make system bootable in UEFI (see "UEFI booting" below)
4: 8300 - Linux filesystem, to store GRUB configuration (grub.cfg) in "/boot" and the GRUB modules to support multiple boot options.
9: BF07 - Solaris Reserved 1

Create LUKS encrypted volume

Create a LUKS volume called: cryptvol

Since you are testing in QEMU, you may use a quick, but less-secure --use-urandom PRNG.

cryptsetup luksFormat --cipher aes-xts-plain64 --key-size 512 --hash sha512 --use-random -y /dev/disk/by-id/virtio-VIRTDISK1-part1
cryptsetup luksOpen /dev/disk/by-id/virtio-VIRTDISK1-part1 cryptvol

Create ZFS pool on an encrypted volume

zpool create -o ashift=12 \
        -O atime=off -O canmount=off -O normalization=formD \
        -O compression=lz4 -O dedup=on -O mountpoint=/ -R /mnt \
        rpool cryptvol

I have used lz4 compression as it has high performance and data deduplication dedup=on to save space.

WARNING: Before enabling deduplication, make sure you understand the memory-related constraints first.

More information on ashift=12 can be found here - ZFS Performance Considerations

Configure ZFS pool and filesystems

zfs create -o canmount=off -o mountpoint=none rpool/ROOT
zpool set bootfs=rpool/ROOT rpool
zfs create -o canmount=noauto -o mountpoint=/         rpool/ROOT/ubuntu
zfs mount                                             rpool/ROOT/ubuntu
zfs create                 -o setuid=off              rpool/home
zfs create -o mountpoint=/root                        rpool/home/root
zfs create -o canmount=off -o setuid=off  -o exec=off rpool/var
zfs create                                            rpool/var/spool
zfs create -o com.sun:auto-snapshot=false -o exec=on  rpool/var/tmp
zfs create -o com.sun:auto-snapshot=false -o exec=on -o mountpoint=/var/lib/docker  rpool/docker

Prepare for chrooting

Bootstrap

Bootstrap Ubuntu 16.04.1 LTS (Xenial)

chmod 1777 /mnt/var/tmp
debootstrap xenial /mnt
zfs set devices=off rpool

You may want to take a snapshot at this point and also during the process if you wish so

zfs snapshot rpool/ROOT/ubuntu@bootstrap
zfs list -t snapshot

Set the hostname

replace vm1 with your hostname

echo vm1 > /mnt/etc/hostname
echo "127.0.1.1       vm1" >> /mnt/etc/hosts

Set the network interace

Do not apply this in case you are going to use only WiFi.

NIF=$(route | grep '^default' | grep -o '[^ ]*$')

cat << EOF > /mnt/etc/network/interfaces.d/$NIF
auto $NIF
iface $NIF inet dhcp
EOF

When using WiFi, you can apply this sequence to get your network started:

wpa_passphrase yourSSID > /root/wifi.conf
wpa_supplicant -B -i wlp4s0 -c /root/wifi.conf
dhclient wlp4s0

Chroot into your future system

mount --bind /dev  /mnt/dev
mount --bind /dev/pts  /mnt/dev/pts
mount --bind /proc /mnt/proc
mount --bind /sys  /mnt/sys
chroot /mnt /bin/bash --login

Configure the system

Create apt sources file

To see current release use lsb_release -cs command.

cat << EOF > /etc/apt/sources.list
deb http://archive.ubuntu.com/ubuntu xenial main universe
deb-src http://archive.ubuntu.com/ubuntu xenial main universe

deb http://security.ubuntu.com/ubuntu xenial-security main universe
deb-src http://security.ubuntu.com/ubuntu xenial-security main universe

deb http://archive.ubuntu.com/ubuntu xenial-updates main universe
deb-src http://archive.ubuntu.com/ubuntu xenial-updates main universe
EOF

Generate mtab file

/etc/mtab should be a symlink pointing to /proc/self/mounts

ln -s /proc/self/mounts /etc/mtab

Generate locales

This adds en_US.UTF-8 UTF-8 and en_GB.UTF-8 UTF-8 to /etc/locale.gen` and generates/usr/lib/locale/locale-archive``.

locale-gen en_US.UTF-8 en_GB.UTF-8

Update global local settings

This adds LANG=en_US.UTF-8 and LC_ALL=en_GB.UTF-8 to /etc/default/locale file.

update-locale LANG=en_US.UTF-8 LC_ALL=en_GB.UTF-8

Set the timezone

This updates /etc/localtime symlink

dpkg-reconfigure tzdata

Create missing groups

addgroup --system lpadmin
addgroup --system sambashare
mkdir /var/cache/samba

/var/cache/samba directory is needed, otherwise gvfsd will complain gvfsd[3251]: mkdir failed on directory /var/cache/samba: Permission denied

Set a password for the root user

passwd

Update software

apt update && apt -y upgrade && apt -y dist-upgrade && apt autoremove
apt -y install ubuntu-minimal

optionally, take a snapshot

zfs snapshot rpool/ROOT/ubuntu@minimal

Prepare for boot

Create boot filesystem

Create a GRUB /boot partition and mount it

mkfs.ext4 -L GRUB2 /dev/disk/by-id/virtio-VIRTDISK1-part4
echo "LABEL=GRUB2  /boot  ext4  defaults  0 2" >> /etc/fstab
mount /boot

Configure system for boot

Add a reference to your encrypted LUKS volume, since this will be required by the update-initramfs tool.

echo "/dev/disk/by-id/dm-name-cryptvol  /  zfs  defaults 0 0" >> /etc/fstab

Make the following symlinks, so that grub-probe / will not fail in any case.

ln -sv /dev/disk/by-id/dm-name-cryptvol /dev/cryptvol
ln -sv /dev/disk/by-id/dm-name-cryptvol /dev/dm-name-cryptvol

Install prerequisites

dmsetup has to be installed along with cryptsetup, otherwise cryptsetup luksOpen will hang on semop syscall.

apt -y install dmsetup cryptsetup zfs-initramfs
apt -y --no-install-recommends install linux-image-generic

You might want to install linux-headers-generic linux-firmware linux-tools-generic along with the linux-image-generic.
But take into account that installing linux-headers-generic might take some time, since it will cause DKMS to rebuild ZFS module, which is likely to happpen when you are installing a different version of Linux headers from currently running Linux kernel version.

Update the crypttab file

Find the UUID of your encrypted partition and reference it in /etc/crypttab file.

# blkid /dev/disk/by-id/virtio-VIRTDISK1-part1 
/dev/disk/by-id/virtio-VIRTDISK1-part1: UUID="85c304e3-58d5-401e-8d8e-a0226195e45b" TYPE="crypto_LUKS" PARTUUID="38b80f0a-b0db-4bac-9dcd-5a6743d9a95e"

You can omit discard if you are not using SSD drives.
Make sure you are aware of security implications using it, refer to man cryptsetup for the details.

echo "cryptvol UUID=85c304e3-58d5-401e-8d8e-a0226195e45b none luks,discard" >> /etc/crypttab

Add the following udev rule in order to produce /dev/cryptvol and /dev/dm-name-cryptvol symlinks pointing to the real opened LUKS device at every system boot.

echo 'ENV{DM_NAME}=="cryptvol", SYMLINK+="cryptvol"' > /etc/udev/rules.d/99-local.rules
echo 'ENV{DM_NAME}=="cryptvol", SYMLINK+="dm-name-cryptvol"' >> /etc/udev/rules.d/99-local.rules

optionally, take a snapshot

zfs snapshot rpool/ROOT/ubuntu@pregrub

Legacy BIOS booting (non-UEFI)

You can skip non-UEFI section in case if you want to use only UEFI mode for booting.

Install the boot loader

apt -y install grub-pc

Install GRUB to the disk(s), not the partition(s).
running grub-probe / should return zfs

Install stage1 GRUB loader to your disk

grub-install /dev/disk/by-id/virtio-VIRTDISK1

UEFI booting

If you want your system to be bootable in UEFI, then apply the following steps:

apt -y install dosfstools
mkdosfs -F 32 -n EFI /dev/disk/by-id/virtio-VIRTDISK1-part3
mkdir /boot/efi
echo PARTUUID=$(blkid -s PARTUUID -o value \
      /dev/disk/by-id/virtio-VIRTDISK1-part3) \
      /boot/efi vfat defaults 0 2 >> /etc/fstab
mount /boot/efi
apt -y install grub-efi-amd64
grub-install --target=x86_64-efi --efi-directory=/boot/efi \
      --bootloader-id=ubuntu --recheck --no-floppy

After that you will have EFI data under /boot/grub/x86_64-efi directory and a /boot/efi/EFI/ubuntu/grubx64.efi file.

Note: if your system isn’t booting, then change BIOS settings to boot with UEFI, then boot Ubuntu Live, apply the troubleshooting steps described below in order to mount your system partition. Then just re-run efi grub-install again.

If the /sys/firmware/efi directory is present, then you are in UEFI mode.
Running efibootmgr -v from chroot, can also give you a hint.

Generate an initramfs image

update-initramfs -c -k all

The contents of initrd should contain cryptsetup binaries.

# lsinitramfs /boot/initrd.img-4.4.0-62-generic |grep -E "cryptsetup$|cryptroot$"
scripts/local-top/cryptroot
scripts/local-block/cryptroot
lib/cryptsetup
sbin/cryptsetup
conf/conf.d/cryptroot

Generate a GRUB configuration file

# vi /etc/default/grub
Comment out: GRUB_HIDDEN_TIMEOUT=0
Remove quiet and splash from: GRUB_CMDLINE_LINUX_DEFAULT
Uncomment: GRUB_TERMINAL=console
Save and quit.

You can use this oneliner

sed -i.bkp -e '/GRUB_HIDDEN_TIMEOUT/ s/^#*/#/' -e '/GRUB_CMDLINE_LINUX_DEFAULT/ s/".*"//' -e '/GRUB_TERMINAL=console/ s/^#//' /etc/default/grub

Generate a /boot/grub/grub.cfg file (used by a stage2 GRUB loader):

update-grub

Check /boot/grub/grub.cfg for a line like this:
linux /vmlinuz-4.4.0-62-generic root=ZFS=rpool/ROOT/ubuntu ro

Prepare to your first boot

Install wpasupplicant in case you are connecting only over WiFi.

apt -y --no-install-recommends install wpasupplicant

optionally, take a snapshot

zfs snapshot rpool/ROOT/ubuntu@preboot

Gracefully exit chroot environment

exit
umount -lf /mnt/boot/efi /mnt/boot /mnt/sys /mnt/proc /mnt/dev/pts /mnt/dev
zpool export rpool
cryptsetup luksClose cryptvol
sync
reboot

First boot

At the first boot make sure

  • /boot and /boot/efi devices are mounted
  • /dev/crypvol and /dev/dm-name-crypvol are valid symlinks
  • grub-probe / returns zfs

optionally, take a snapshot

zfs snapshot rpool/ROOT/ubuntu@firstboot

Install OpenSSH server if you wish to continue installation remotely.

apt -y install openssh-server

Create a user

You can create a separate ZFS FS for your user

zfs create rpool/home/YOURUSER

Create your user and set its password

useradd -s /bin/bash -G adm,cdrom,sudo,dip,plugdev,lpadmin,sambashare YOURUSER
cp -p -- /etc/skel/.??* /home/YOURUSER/
passwd YOURUSER
chown -Rh YOURUSER:YOURUSER /home/YOURUSER
chmod 0750 /home/YOURUSER

Configure SWAP

change 256M to 4G or more if you have more space, depending on your needs.

zfs create -V 256M -b $(getconf PAGESIZE) -o compression=zle \
      -o logbias=throughput -o sync=always \
      -o primarycache=metadata -o secondarycache=none \
      -o com.sun:auto-snapshot=false rpool/swap

Now you can format and activate swap

mkswap -f /dev/zvol/rpool/swap 
echo "/dev/zvol/rpool/swap none swap defaults 0 0" >> /etc/fstab
swapon -av

Install Desktop environment

apt -y install ubuntu-desktop

If you are going to use NetworkManager then don’t forget to remove previously created configuration under /etc/network/interfaces.d/ directory.

Extra tips

Since you are now using ZFS with enabled compression, you can remove redundant compression option in configuration files under /etc/logrotate.d/ directory.

Since everything is working well and you wish to release occupied space, you might want to remove previously created ZFS snapshots, e.g. zfs destroy rpool/ROOT/ubuntu@install.

Disable root password passwd -d root.

Useful packages to install: bash-completion rsync dnsutils whois lsof rng-tools acpi sysstat

Troubleshooting

One day you may want to boot Live Linux and mount your system partitions.

Below are the steps to do it properly.

apt -y install debootstrap gdisk zfsutils-linux dmsetup cryptsetup
zpool export -a
cryptsetup luksOpen /dev/disk/by-id/virtio-VIRTDISK1-part1 cryptvol
zpool import -N -R /mnt -d /dev/disk/by-id/ rpool
zpool status
zfs mount rpool/ROOT/ubuntu
zfs mount -a

mount --bind /dev  /mnt/dev
mount --bind /dev/pts  /mnt/dev/pts
mount --bind /proc /mnt/proc
mount --bind /sys  /mnt/sys
chroot /mnt /bin/bash --login
mount /boot
mount /boot/efi

When done fixing your system, gracefully exit chroot environment

exit
umount -lf /mnt/boot/efi /mnt/boot /mnt/sys /mnt/proc /mnt/dev/pts /mnt/dev
zpool export rpool
cryptsetup luksClose cryptvol
sync
reboot


The postings on this site are my own and don't necessarily represent my employer’s positions, strategies or opinions.

© Andy, 2022 — built with Jekyll using Lagom theme