Ubuntu NAS - ZFS on the HP Microserver

HP Microserver N40L

I had loads of fun setting up FreeNAS on a HP Microserver and can highly recommend this combination for a small home/office file server. But I had a few problems mainly with the performance and stability of the iScsi driver that comes with it. So I decided to pull out the FreeNAS/freebsd root disk and put in a new one on which to check out Ubuntu 12.04 server in combination with the native linux ZFS filesystem.

My target was to get 1Gb/s read and write performance out of the little microserver - ideally iScsi, but NFS would be OK too. Amazingly although the former appears to be a little optimistic the latter is acheivable.

I had setup FreeNAS with 3x2TB disks in a RAIDZ1 array. I replaced the root disk with a new one leaving the 3 ZFS disks in situ and installed ubuntu 12.04 server on the fresh root disk. I created a 'nas' user account for the box and got the installer to put SSH on for me, but nothing else.

ZFS installation is straightforward - i installed bonnie++ too so I could do some benchmarks on the box.

# apt-get update 

# apt-get upgrade 

# apt-get install python-software-properties 

# apt-add-repository ppa:zfs-native/stable 

# apt-get update 

# apt-get install ubuntu-zfs nfs-server sysstat iscsitarget iscsitarget-dkms open-iscsi bonnie

Once installed mounting the existing ZFS array was a case of doing

 # zfs import -f nasvol

and /nasvol appeared by magic on the root of the filesystem.

I then set about benchmarking performance of ZFS, ext4 (the root filesystem) NFS, iScsi using bonnie++. The fastest combination turned out to be a 4 drive RAID10 ZFS setup with NFS. I couldn't get iScsi to perform without large latency, i/o coming in bursts or high CPU. NFS on the other hand was pretty sweet - locally managing 180MB/s write and 250MB/s read. NFS speeds top out at 96MB/s write and 105MB/s read.

To get this performance, a few tips:

1. ZFS performance is woeful (40MB/s) unless you create the zpool with '-o ashift=12'. This makes it use 4k blocks (i think) which coincides with the modern SATA drives.
2. More spindles, more speed. Raid 10 gives it 4 to read from and 2 to spread the writes accross. 
3. RAIDZ1 is great, but not with the Microserver's horsepower. Get an i5/i7 and you'll be able to enjoy deduplication too!
4. NFS server set up rw, async, no_subtree_check, wdelay. (The UPS is on order)
5. NFS client set up to use rsize/wsize 65536, tcp, and noatime.
6. It's possible to disable checksumming in ZFS, but it didn't make much different in this setup.

As I mentioned above, iScsi performance didn't really compare too well with NFS. My ZFS zVol iScsi target managed 70MB/s writes and 50MB/s reads. None of the combinations of LVM2/ext4/iScsi I tried managed more than 80MB/s write although some did manage 100MB/s read.