Aidan Harris - FreeBSD Storage Server on Libvirt Aidan Harris - FreeBSD Storage Server on Libvirt

This week I suffered a catastrophic storage failure that was entirely my fault! Needless to say this is somewhat annoying. On the other hand this gives me a perfect opportunity to check out FreeBSD and ZFS.

Getting the Installation Media

This couldn't be easier. The fine folks over at freebsd.org have virtual machine images available. Since we're using libvirt we'll want the qcow2 file.

Getting Going

I'm not going to talk too much about "getting going" since this is ridiculously easy, even more so if you are using virt-manager. If you're using virt-manager simply point it at the qcow2, set the CPU and RAM, boot and you're done!

First Steps

Updating the system

FreeBSD uses the pkg package manager. To look for updates simply run pkg update to apply any needed upgrades run pkg upgrade.

Allowing serial-port access

By allowing serial access to FreeBSD you can easily get a login shell using virsh console freebsd11.0 (where freebsd11.0 is the name of your VM).

echo 'console="comconsole"' >> /boot/loader.conf

Now if you restart the VM (shutdown -r now) you should be able to connect to it in a new terminal via sudo virsh console freebsd11.0.

Auto-start VM

sudo virsh autostart freebsd11.0

Creating Users

Adding a users group

Before creating a user we should create a "users" group to house them in. This is consistent with Arch Linux and will give us less problems down the line as long as the UIDs and GIDs match up.

To create a users group with a GID of 100 use the following command:

pw groupadd -n users -g 100

Adding a user

pw useradd -n user -g users -m # Create a user named 'user', with a primary group 'users' and a home directory '/home/user' which is created for you

Configuring Storage

Block Device Pass-through

ZFS likes working with raw disks, luckily we can pass them through to our FreeBSD guest. RedHat has excellent documentation on this here.

For testing purposes you can use the following command to attach a block device:

virsh attach-disk freebsd11.0 /dev/sr0 vdc

After executing that command you should see the disk in the FreeBSD guest:

sudo virsh console freebsd11.0
[email protected]:~ # gpart show
=>       3  44040315  vtbd0  GPT  (21G)
         3       118      1  freebsd-boot  (59K)
       121   2097152      2  freebsd-swap  (1.0G)
   2097273  41943040      3  freebsd-ufs  (20G)
  44040313         5         - free -  (2.5K)

=>        34  3907029101  vtbd1  GPT  (1.8T)
          34  3907029101         - free -  (1.8T)

To make this permanent you can edit the xml configuration:

sudo -E virsh edit freebsd11.0

Add the following underneath the first <disk> entry (there should only be one <disk> in the entire file which should be our qcow2 boot medium):

<disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source dev='/dev/sr0'/>
      <target dev='vdc' bus='virtio'/>
</disk>

Now if you reboot the vm (sudo virsh reboot freebsd11.0) and issue a gpart show again, you should see the disk.

Note: The Linux kernel makes no guarantees over the disk ordering / labelling. Do Not passthrough a disk via it's alphabetic identifier (e.g /dev/sda) instead use it's id (/dev/disk/by-id) or uuid (/dev/disk/by-uuid) which should be unique.

ZFS

I've not used ZFS before but the FreeBSD Handbook has a "Quick Start Guide" here.

For reference I'm working with 3x2tb drives which I intend to use in a Raid5 (RaidZ?) configuration.

Enable ZFS

echo 'zfs_enable="YES"' >> /etc/rc.conf
service zfs start

Create zpool

zpool create storage raidz vtbd1 vtbd2 vtbd3

To verify that the zpool has been created you can issue the command zpool list:

[email protected]:~ # zpool list
NAME      SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
storage  5.44T   161K  5.44T         -     0%     0%  1.00x  ONLINE  -

Create Storage

zfs create storage/home
zfs set copies=2 storage/home
zfs set compression=lz4 storage/home

Hammer Time

At this point I'm already in love with ZFS but now it's time to fill this sucker with data and see what it's really got:

rsync -av --progress [email protected]:/tmp/gparted-roview-2DZ7RV/Music /storage/home

After transfering data I ran du -h /storage/home/Music which size reports exactly double the size on the source disk (presumably because we ran zfs set copies=2 storage/home). It's nice that du tells me this since other filesystems such as btrfs on Linux often report disk usage incorrectly (yes, I know you can fix this with the btrfs filesystem du and btrfs filesystem df commands).

SSHD

echo 'sshd_enable="YES"' >> /etc/rc.conf
service sshd start

NFS

Detailed information regarding NFS can be found in the FreeBSD Handbook here.

echo '/storage/home  -alldirs  192.168.x.x' >> /etc/exports
echo 'rpcbind_enable="YES"' >> /etc/rc.conf
echo 'nfs_server_enable="YES"' >> /etc/rc.conf
echo 'mountd_flags="-r"' >> /etc/rc.conf
service nfsd start

Samba

Detailed information regarding Samba can be found in the FreeBSD handbook here.

pkg install samba46
echo 'samba_enable="YES"' >> /etc/rc.conf
echo 'samba_server_enable="YES"' >> /etc/rc.conf
service samba_server start

Modify the following samba at /usr/local/etc/smb4.conf for your needs:

[global]
workgroup = WORKGROUP
server string = Samba Server Version %v
netbios name = ExampleMachine
wins support = Yes
security = user
passdb backend = tdbsam

# Example: share /usr/src accessible only to 'developer' user
[src]
path = /usr/src
valid users = developer
writable  = yes
browsable = yes
read only = no
guest ok = no
public = no
create mask = 0666
directory mask = 0755