FreeNAS: Convert an Encrypted Disk to Mirror

This article will explain how to convert a single disk vdev in an encrypted ZFS storage pool to a mirror. This is useful if you have a single disk vdev and want to add redundancy to handle a drive failure. If you are setting up a vdev for the first time and have 2 disks available, you don’t need this.

I wanted to migrate some drives to my FreeNAS without taking the origin array offline. In order to do this I setup a new encrypted ZFS volume (pool) on FreeNAS with a single disk vdev. After I moved some data over (and with a good backup), I was able to pull a 2nd drive from the origin array and add it to FreeNAS, eventually using it as a mirror to add redundancy to the pool.

NOTE: This is the process I generally followed, but you may want to experiment in a VM before working with live data.

Key Terms

• ZFS storage pool: A pool of vdevs. FreeNAS refers to this as a “volume”.
• vdev: A virtual storage device (disk, file, mirror, raidz, spare, cache, or log)
• geli: A block-level encryption system for FreeBSD which operates below the ZFS layer in FreeNAS

Overview

The high-level steps are:

1. Add the new disk to the system
2. Parition the disk with gpart
3. Encrypt the data parition using geli and the existing volume / pool key
4. Attach the new encrypted parition to ZFS as a mirror
5. Set and backup the recovery key
6. Update the FreeNAS encrypted disk list

Unfortunately, after performing steps 1–5, rebooting, and unlocking the pool, FreeNAS will not attempt to unlock the second disk with geli and zpool status will report it as unavailable. This can be resolved before reboot using one of the following methods to update the encrypted disk list, so you’ll have to decide which approach you want to take.

• Method 1: Export and reimport the pool so FreeNAS updates its disk list. This should be the simplest option but I haven’t tested it. After performing steps 1–5, make sure your encryption and recovery keys are backed up. Next, in the GUI, select the pool, click Detach Volume (but obviously don’t use the options to delete your data or shares), then use Auto Import to restore the pool. This correlates to zpool export and zpool import.
• Method 2: Attach to ZFS, abort resilvering, replace via the GUI. That is, after attaching the encrypted partition in step #4 above, immediately take the disk offline using zpool offline <pool> <dev> to abort resilvering. The vdev is now in mirror configuration and so you can use the normal Replace disk feature in the FreeNAS GUI. It will repartition the disk, setup the encryption again, and begin resilvering, but this time FreeNAS will remember the disk is encrypted. The FreeNAS docs recommend performing a re-key, password change, new recovery key, and backup after this although it seems like just setting the recovery key should be all that is required.
• Method 3: If you already rebooted and the pool shows the new disk as unavailable you can either follow Method #2 by replacing the disk using the GUI and going through resilvering again or you can manually attach the disk from the shell using geli attach and zpool online followed by Method #1.

Now that we’ve covered the high-level plan, here are the details.

Partition the Disk

Create the GPT

Use the GUI to find the device ID of the new drive or usecamcontrol.

# camcontrol devlist
<WDC WD20EARX-000000 51.0AB51>     at scbus4 target 0 lun 0 (ada0,pass0)        
<WDC WD20EARX-000000 51.0AB51>    at scbus5 target 0 lun 0 (ada1,pass1)        
<WDC WD20EARS-000000 51.0AB51>    at scbus6 target 0 lun 0 (ada2,pass2)        
<Flash Drive SM_USB20 1100>        at scbus10 target 0 lun 0 (da0,pass3)

Create the GUID partition table.

# gpart create -s gpt ada1
ada1 created

Add the Swap Partition

Create a swap partition matching what FreeNAS created on the original drive. FreeNAS puts a swap partition on every data drive by default, stripes them together, and encrypts with a temporary key each boot. I’m not sure how that works when a drive fails, but it’s the recommended configuration.

# gpart add -b 128 -i 1 -t freebsd-swap -s 2G ada1
ada1p1 added

One advantage to a swap partition is it leaves you with extra blocks if you need to borrow some to mirror using a drive that isn’t precisely the same size. Swap creation isn’t mandatory and can be disabled in the System/Settings/Advanced GUI view by setting the swap size to 0. You can also use swapon and swapoff to update swap settings.

Add the Data Partition

Use the remaining space for the data partition.

# gpart add -i 2 -t freebsd-zfs ada1
ada1p2 added

Encrypt the Data Partition

Get the GPTID for the Partition

A device may change names depending on the connected port but the GPTID doesn’t change. FreeNAS uses the GPTID to track disks and so we want the rawuuid field of ada1p2.

# gpart list

Geom name: ada1
...
scheme: GPT
Providers:
1. Name: ada1p1
   Mediasize: 2147483648 (2.0G)
   ...
   rawuuid: de0921a2-2b95-11e3-94b2-0023543761d8
   ...
2. Name: ada1p2
   Mediasize: 1998251364352 (1.8T)
   ...
   rawuuid: e9bffad5-2b95-11e3-94b2-0023543761d8
   ...

Encrypt the Data Partition

Use geli to initialize the data partition with a sector size of 4096, no metadata backups, the pre-existing key for the pool, the target partition in GPTID format, and prompt for the passphrase.

You can find your pool key in the /data/geli folder or use a copy you previously backed up. Note that this is not the same as the recovery key, nor is it the full key used to secure the master encryption key for the disk. This key plus the passphrase is required to unlock the master key for decryption.

# geli init -s 4096 -B none -K /data/geli/7ded15b2-3a67-1c1d-4a08-f44f1eda9c70.key /dev/gptid/e9bffad5-2b95-11e3-94b2-0023543761d8

This initializes the geli metadata for the partition and sets the combined key + passphrase into key slot 0. You can add a recovery key to slot 1 using geli setkey or later using the GUI once the partition is added to the ZFS pool.

Attach the Encrypted Partition

Attach using the key and passphrase.

# geli attach -k /data/geli/7ded15b2-3a67-1c1d-4a08-f44f1eda9c70.key /dev/gptid/e9bffad5-2b95-11e3-94b2-0023543761d8
Enter passphrase: ...

Verify the attach and note the .eli extension for the encrypted partition.

# geli status
                                          Name  Status  Components
                                    ada0p1.eli  ACTIVE  ada0p1
gptid/95f8c4e9-2739-11e3-bfda-0023543761d8.eli  ACTIVE  gptid/95f8c4e9-2739-11e3-bfda-0023543761d8
gptid/ad2c2c00-2739-11e3-bfda-0023543761d8.eli  ACTIVE  gptid/ad2c2c00-2739-11e3-bfda-0023543761d8
gptid/e9bffad5-2b95-11e3-94b2-0023543761d8.eli  ACTIVE  gptid/e9bffad5-2b95-11e3-94b2-0023543761d8

Attach to ZFS

Attach the encrypted partition using zpool which will begin the resilvering process. You will need the GPTID of the encrypted original disk parition.

# zpool attach <pool> <original disk> <new disk>

# zpool attach pool1 /dev/gptid/95f8c4e9-2739-11e3-bfda-0023543761d8.eli /dev/gptid/e9bffad5-2b95-11e3-94b2-0023543761d8.eli

You can check the status of the resilvering process by using zpool status.

# zpool status
  pool: pool1
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Wed Oct  2 13:11:44 2013
        418M scanned out of 769G at 32.1M/s, 6h48m to go
        414M resilvered, 0.05% done
config:

    NAME                                                STATE     READ WRITE CKSUM
    pool1                                               ONLINE       0     0     0
      mirror-0                                          ONLINE       0     0     0
        gptid/95f8c4e9-2739-11e3-bfda-0023543761d8.eli  ONLINE       0     0     0
        gptid/e9bffad5-2b95-11e3-94b2-0023543761d8.eli  ONLINE       0     0     0  (resilvering)

Set the Recovery Key

Your mirror is now online and you can use the GUI to set the recovery key. I did not verify that the recovery key was set, and since FreeNAS may not have an up to date disk list, it may be safest to perform this step again after updating the disk list. You can also take the disk offline with zpool offline and geli detach / attach if you want to test your recovery key before rebooting.

Update the FreeNAS Encrypted Disk List

After resilvering is complete, and as mentioned in the overview, use the GUI to detach and then auto import the encrypted pool. Or, try one of the other methods.

Conclusion

If you are familiar with zpool and geli then this manual process to attach a mirror isn’t too painful; the main tripping point is getting FreeNAS to see the newly encrypted disk on reboot. There are FreeNAS feature requests to incorporate this process into the GUI so you may want to watch those for future updates.

• Feature #3128: Ability to attach mirrors from GUI
• Feature #3191: GUI should allow attaching disk as mirror of existing one

I hope this helped!