November 05, 2013

TRIM in Linux with SSD disk, LVM, and encryption

A Trim command (commonly typeset as TRIM ) allows an operating system to inform a solid-state drive (SSD) which blocks of data are no longer considered in use and can be wiped internally.

Trimming enables the SSD to handle garbage collection overhead, which would otherwise significantly slow down future write operations to the involved blocks, in advance.

In this article I’m writing how to enable and use TRIM with your SSD disk.

Make sure TRIM is supported by your SSD disk

root@homepc:~# hdparm -I /dev/sda |grep -i trim
	   *	Data Set Management TRIM supported (limit 1 block)
	   *	Deterministic read data after TRIM

Block devices

root@homepc:/tmp# lsblk -D /dev/sda
NAME                     DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO
sda                             0      512B       2G         0
├─sda1                          0      512B       2G         0
├─sda2                          0      512B       2G         0
└─sda5                          0      512B       2G         0
  ├─rootvg-root (dm-0)          0      512B       2G         0
  ├─rootvg-swap_1 (dm-1)        0      512B       2G         0
  ├─rootvg-usr (dm-2)           0      512B       2G         0
  ├─rootvg-var (dm-3)           0      512B       2G         0
  ├─rootvg-tmp (dm-4)           0      512B       2G         0
  └─rootvg-home (dm-5)          0      512B       2G         0

Disk partition table

root@homepc:/tmp# fdisk  -u -l /dev/sda

Disk /dev/sda: 60.0 GB, 60022480896 bytes
255 heads, 63 sectors/track, 7297 cylinders, total 117231408 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0003d3db

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048      499711      248832   83  Linux
/dev/sda2          501758   117229567    58363905    5  Extended
/dev/sda5          501760   117229567    58363904   8e  Linux LVM

Add “discard” to a FS that you want to have TRIM enabled for

root@homepc:/tmp# grep -w /tmp /etc/fstab
/dev/mapper/rootvg-tmp /tmp            ext4    defaults,discard,noatime        0       2

root@homepc:~# mount -o remount /tmp

root@homepc:/tmp# mount |grep -w /tmp
/dev/mapper/rootvg-tmp on /tmp type ext4 (rw,noatime,user_xattr,barrier=1,data=ordered,discard)

root@homepc:~# df -h /tmp
Filesystem              Size  Used Avail Use% Mounted on
/dev/mapper/rootvg-tmp  368M   11M  339M   3% /tmp

Following option lets LVM to issue discards only to a LV that is no longer using the PV (e.g. lvremove, lvreduce, etc) see ‘man lvm.conf’ for issue_discards

root@homepc:/tmp# grep -w issue_discards /etc/lvm/lvm.conf
    issue_discards = 1

Create 16bytes file filled with any data (in this example I filled it with “1”)

root@homepc:/tmp# while true; do printf 1;done |dd bs=16 count=1 of=./test.txt
1+0 records in
1+0 records out
16 bytes (16 B) copied, 2.7302e-05 s, 586 kB/s
root@homepc:/tmp# cat ./test.txt  ; echo
1111111111111111

Check file offset relative to a FS’s underlying LV

root@homepc:/tmp# filefrag -bsv ./test.txt
Filesystem type is: ef53
File size of ./test.txt is 16 (1 block, blocksize 1024)
 ext logical physical expected length flags
   0       0     8752               1 eof
./test.txt: 1 extent found

Note To keep the same phyical address (when doing filefrag -bsv) just remove the file, sync and drop caches; then after you can recreate it and it will have the same physical address

rm /tmp/test.txt
sync && echo 1 | tee /proc/sys/vm/drop_caches

while true; do printf 6;done |dd bs=512 count=1 of=/tmp/test.txt
sync && echo 1 | tee /proc/sys/vm/drop_caches
hexdump -C /tmp/test.txt
filefrag -bsv /tmp/test.txt

Checking LV’s sector offset relative its underlying PV

root@homepc:/tmp# dmsetup table
rootvg-tmp: 0 778240 linear 8:5 35678208
rootvg-usr: 0 17571840 linear 8:5 681984
rootvg-var: 0 5857280 linear 8:5 18253824
rootvg-home: 0 80265216 linear 8:5 36456448
rootvg-swap_1: 0 11567104 linear 8:5 24111104
rootvg-root: 0 679936 linear 8:5 2048

Free pagecache in order to make sure that the data has been physically written on a disk

root@homepc:/tmp# sync && mount -o remount /tmp && echo 1 | tee /proc/sys/vm/drop_caches
1

More info on drop_caches you can find in the official sysctl vm documentation

Reading ‘test.txt’ file contents from its underlying LV and disk (using hexdump and dd)

root@homepc:/tmp# hexdump -C -s $((8752*1024)) -n 1024 /dev/mapper/rootvg-tmp
0088c000  31 31 31 31 31 31 31 31  31 31 31 31 31 31 31 31  |1111111111111111|
0088c010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
0088c400

root@homepc:/tmp# dd if=/dev/sda bs=512 count=1 skip=$(( 501760+(35678208)+(8752*1024)/512 )) status=noxfer 2>/dev/null|hexdump -C
00000000  31 31 31 31 31 31 31 31  31 31 31 31 31 31 31 31  |1111111111111111|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000200

Discard unused blocks on a mounted filesystem

root@homepc:/tmp# fstrim -v /tmp/
/tmp/: 372541440 bytes were trimmed

Make sure that unused blocks were really discarded

root@homepc:/tmp# dd if=/dev/sda bs=512 count=1 skip=$(( 501760+(35678208)+(8752*1024)/512 )) status=noxfer 2>/dev/null|hexdump -C
00000000  31 31 31 31 31 31 31 31  31 31 31 31 31 31 31 31  |1111111111111111|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000200

Looks that nothing happened… However

root@homepc:/tmp# sync && mount -o remount /tmp && echo 1 | tee /proc/sys/vm/drop_caches
1

root@homepc:/tmp# dd if=/dev/sda bs=512 count=1 skip=$(( 501760+(35678208)+(8752*1024)/512 )) status=noxfer 2>/dev/null|hexdump -C
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000200

TRIM worked!

LUKS encryption

If you have encrypted PV, then make sure you have specified “discard” in your /etc/crypttab file

root@homepc:~# grep discard /etc/crypttab
sdb5_crypt UUID=7a4cd5bf-d188-4885-8a87-d981f4fc32fe none luks,discard

You’ll also need to update your initramfs

root@homepc:~# update-initramfs -u -k all

Reboot and check

root@homepc:~# dmsetup table /dev/mapper/sdb5_crypt
0 116723712 crypt aes-xts-plain64 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 0 8:5 4096 1 allow_discards

Make sure you have the ‘allow_discards’ in the end of the line (scroll to the right)

WARNING There are several security consequences, please read at least http://asalor.blogspot.com/2011/08/trim-dm-crypt-problems.html before you enable it.

To find ‘test.txt’ file contents on an encrypted PV, you can refer to “Physical layout of data on disk and encryption in Linux” article

IMPORTANT Many modern SSDs will not reclaim the TRIMmed space. For this reason you may not see zeros at the end of the test even though TRIM could be working. This weakens the reliability of the test when showing that TRIM is not working. However if you see zeros at the end, then TRIM is definitely working. To put it simply - if the test tells you that TRIM is working (zeros show up), then TRIM is working; if the test tells you that TRIM may not be working (zeros do not show up), then TRIM may or may not be working. In the latter case, simply double check that you have used the right mount options in /etc/fstab and if they look correct, TRIM most probably works. Simply your SSD is not reclaiming the space immediately, so nothing to worry about.