Enabling Software RAID and Disk Encryption on a Hetzner dedicated machine

// Publicado em: 28 de março de 2023

We rented a dedicated machine at Hetzner to run uncloud.do, but the machine has no disk encryption. Let’s fix that.

Investigating

The machine came with Ubuntu 20.04 by default, using two disks and RAID 1 by software. Let’s take a look on how that looks like:

$ lsblk

NAME    MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sda       8:0    0  5.5T  0 disk  
├─sda1    8:1    0   32G  0 part  
│ └─md0   9:0    0   32G  0 raid1 [SWAP]
├─sda2    8:2    0    1G  0 part  
│ └─md1   9:1    0 1022M  0 raid1 /boot
├─sda3    8:3    0    2T  0 part  
│ └─md2   9:2    0    2T  0 raid1 /
├─sda4    8:4    0  3.5T  0 part  
│ └─md3   9:3    0  3.5T  0 raid1 /home
└─sda5    8:5    0    1M  0 part  
sdb       8:16   0  5.5T  0 disk  
├─sdb1    8:17   0   32G  0 part  
│ └─md0   9:0    0   32G  0 raid1 [SWAP]
├─sdb2    8:18   0    1G  0 part  
│ └─md1   9:1    0 1022M  0 raid1 /boot
├─sdb3    8:19   0    2T  0 part  
│ └─md2   9:2    0    2T  0 raid1 /
├─sdb4    8:20   0  3.5T  0 part  
│ └─md3   9:3    0  3.5T  0 raid1 /home
└─sdb5    8:21   0    1M  0 part

$ sudo mdadm --detail /dev/md0

/dev/md0:
           Version : 1.2
     Creation Time : Mon Feb  7 19:32:29 2022
        Raid Level : raid1
        Array Size : 33520640 (31.97 GiB 34.33 GB)
     Used Dev Size : 33520640 (31.97 GiB 34.33 GB)
      Raid Devices : 2
     Total Devices : 2
       Persistence : Superblock is persistent

       Update Time : Sun Mar  6 03:39:18 2022
             State : clean 
    Active Devices : 2
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 0

Consistency Policy : resync

              Name : rescue:0
              UUID : dea498f4:ab3f2753:705218f5:ecaeb1b0
            Events : 27

    Number   Major   Minor   RaidDevice State
       0       8       17        0      active sync   /dev/sdb1
       1       8        1        1      active sync   /dev/sda1

All /dev/mdX look the same (excluding the size and partition names).

$ sudo cat /proc/mdstat
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] 
md2 : active raid1 sda3[1] sdb3[0]
      2111699968 blocks super 1.2 [2/2] [UU]
      bitmap: 2/16 pages [8KB], 65536KB chunk

md3 : active raid1 sda4[1] sdb4[0]
      3713953344 blocks super 1.2 [2/2] [UU]
      bitmap: 0/28 pages [0KB], 65536KB chunk

md1 : active raid1 sda2[1] sdb2[0]
      1046528 blocks super 1.2 [2/2] [UU]
      
md0 : active raid1 sdb1[0] sda1[1]
      33520640 blocks super 1.2 [2/2] [UU]

$ sudo fdisk -l

Disk /dev/sda: 5.47 TiB, 6001175126016 bytes, 11721045168 sectors
Disk model: HGST HUS726060AL
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 09C3EA88-60C2-42A6-9694-BB417FA369E1

Device          Start         End    Sectors  Size Type
/dev/sda1        4096    67112959   67108864   32G Linux RAID
/dev/sda2    67112960    69210111    2097152    1G Linux RAID
/dev/sda3    69210112  4292874239 4223664128    2T Linux RAID
/dev/sda4  4292874240 11721045134 7428170895  3.5T Linux RAID
/dev/sda5        2048        4095       2048    1M BIOS boot

Partition table entries are not in disk order.


Disk /dev/sdb: 5.47 TiB, 6001175126016 bytes, 11721045168 sectors
Disk model: HGST HUS726060AL
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 02B08673-F358-4FC2-BECF-C28DE0AFEB5C

Device          Start         End    Sectors  Size Type
/dev/sdb1        4096    67112959   67108864   32G Linux RAID
/dev/sdb2    67112960    69210111    2097152    1G Linux RAID
/dev/sdb3    69210112  4292874239 4223664128    2T Linux RAID
/dev/sdb4  4292874240 11721045134 7428170895  3.5T Linux RAID
/dev/sdb5        2048        4095       2048    1M BIOS boot

Partition table entries are not in disk order.


Disk /dev/md0: 31.99 GiB, 34325135360 bytes, 67041280 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/md1: 1022 MiB, 1071644672 bytes, 2093056 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/md3: 3.47 TiB, 3803088224256 bytes, 7427906688 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/md2: 1.99 TiB, 2162380767232 bytes, 4223399936 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

$ cat /etc/fstab

proc /proc proc defaults 0 0
# /dev/md/0
UUID=4a4336ec-e82b-4dcb-bcb1-e8a09394ee2a none swap sw 0 0
# /dev/md/1
UUID=91f1c818-33ad-4b9d-a5e5-25eeb1704fb6 /boot ext3 defaults 0 0
# /dev/md/2
UUID=52306702-5231-49cd-9aa1-4e5a89c9af43 / ext4 defaults 0 0
# /dev/md/3
UUID=54c2bdbe-8b4c-48f5-9379-4a379a9df14e /home ext4 defaults 0 0

It seems we have:

Two disks of 6TB.
Both used in RAID 1.
Four mounting points:
- /
- /home
- /boot (not on RAID!)
- swap

Going for it!

The process is based on the How to install Ubuntu 20.04 with full disk encryption tutorial.

Active the rescue system on Hetzner platform.

SSH into the machine with the given password.

Copy your SSH key into the rescue system with:

$ scp <your-public-ssh-key> root@<ip>:/tmp/authorized_keys

Create the configuration file for installimage, store it at /tmp/setup.conf. It is based on the article mentioned above.

# add your password here!
CRYPTPASSWORD your-encrypt-password

DRIVE1 /dev/sda
DRIVE2 /dev/sdb

SWRAID 1
SWRAIDLEVEL 1

HOSTNAME uncloud1

# the machine didn't boot when I added an encrypted `swap` partition
# so I left it out
PART /boot ext4 1024M
PART / ext4 all crypt

IMAGE /root/images/Ubuntu-2004-focal-64-minimal.tar.gz

SSHKEYS_URL /tmp/authorized_keys

Create the /tmp/post-install.sh script (content from Hetzner article):

#!/bin/bash


add_rfc3442_hook() {
  cat << EOF > /etc/initramfs-tools/hooks/add-rfc3442-dhclient-hook
#!/bin/sh

PREREQ=""

prereqs()
{
        echo "\$PREREQ"
}

case \$1 in
prereqs)
        prereqs
        exit 0
        ;;
esac

if [ ! -x /sbin/dhclient ]; then
        exit 0
fi

. /usr/share/initramfs-tools/scripts/functions
. /usr/share/initramfs-tools/hook-functions

mkdir -p \$DESTDIR/etc/dhcp/dhclient-exit-hooks.d/
cp -a /etc/dhcp/dhclient-exit-hooks.d/rfc3442-classless-routes \$DESTDIR/etc/dhcp/dhclient-exit-hooks.d/
EOF

  chmod +x /etc/initramfs-tools/hooks/add-rfc3442-dhclient-hook
}


# Install hook
add_rfc3442_hook

# Copy SSH keys for dropbear
mkdir -p /etc/dropbear-initramfs
cp -a /root/.ssh/authorized_keys /etc/dropbear-initramfs/authorized_keys

# Update system
apt-get update >/dev/null
apt-get -y install cryptsetup-initramfs dropbear-initramfs

Execute the following command to install the operational system:

$ installimage -a -c /tmp/setup.conf -x /tmp/post-install.sh

Reboot:

$ reboot

After the reboot, log in with SSH into the BusyBox SSH server used to unlock the disk:

$ ssh root@<ip>

Enter the command below and the disk password:

$ cryptroot-unlock

The SSH session will terminate automatically, and the machine will boot up.