I spent some time at Christmas upgrading my home lab in preparation for the new VCAP exams which are due out in the first quarter of 2012. In particular I needed to improve my shared storage and hoped that I could reuse old h/w instead of buying something new. I’ve been using an Iomega IX2-200 for the last year but it’s performance is pretty pitiful so I usually reverted to local storage which rather defeated the purpose.
I started off having a quick look around at my storage options for home labs;
- Hardware appliances from QNAP, Synology, Iomega etc. There’s a great, comprehensive list at SmallNetBuilder.com which includes both performance and cost comparisons.
- Software appliances such as OpenFiler, FreeNAS, Datacore’s SANSymphony/SAN Melody, Starwind’s iSCSI SAN and Nexenta’s Community Edition.
- Virtual appliances such as UberVSA, LeftHand, Netapp’s ONTAP Simulator (Netapp customer’s only and capacity limited)
- Or you can spin your own using a variety of base OSs: Oracle Solaris 11, Oracle Solaris 11 Express (free for non-commercial), FreeBSD (older version of ZFS), OpenIndiana and the upcoming Illumian project which is a fork from the now discontinued OpenSolaris. An interesting project is napp-it.org which shows you how to build your own NAS server using ZFS – well worth a look! Jimmy Dansbo has just published his ‘Poor man’s storage appliance‘ which also looks very interesting and has an .OVA available for quick deployment.
Why pick Nexenta?
I’d used OpenFiler and FreeNAS before (both are very capable) but with so much choice I didn’t have time to evaluate all the other options (Greg Porter has a few comments comparing OpenFiler vs Nexenta). Datacore and Starwind’s solutions rely on Windows rather than being bare metal (which was my preference) and I’ve been hearing positive news about Nexenta more and more recently.
On the technical front the SSD caching and VAAI support make Nexenta stand out from the crowd.
What does Nexenta provide?
- An easy to setup* GUI based appliance which runs on standard x86 hardware (either bare metal or as a VM)
- It uses an advanced filesystem, ZFS, under the hood. ZFS features data integrity (RAID equivalents), compression, deduplication, snapshots and more. Check out this post from Theron Conroy on the features of Nexenta.
- Nexenta (via ZFS) offers various cache options which increase performance – combined with SSDs it’s potentially a killer feature! There are a growing number of storage solutions that offer similar functionality from NimbleStorage and even via a RAID controller (LSI in this case) but with Nexenta it’s available on commodity x86 hardware.
- VAAI support. This got my interest when I first read it – a free software appliance that supports VAAI? That’s got to be worth checking out! Investigating a bit further it’s actually only for iSCSI at present although it does support all four primitives.
What are the different editions of Nexenta?
- NexentaStor Enterprise (nexenta.com). As the name implies this is the ‘big boys’ version which you’re not going to run in your home lab unless money is no object, both from a hardware and licensing perspective. There’s a 45 day trial which is free to download. NOTE: This is not the same as the Community Edition!
- NexentaStor Community Edition (known as CE, nexentastor.org). This is what I’m running as it’s free to use for test/dev and has an 18TB capacity restriction. Not a problem for my 2TB home lab!
- NexentaStor Core (CLI only, nexenta.org). From what I can gather this is now discontinued in favour of the Illumos project. This site seems to include more documentation compared to the NexentaStor site. For base OS and general ZFS queries this site seems more useful.
As of Feb 2012 the latest version is 3.1.2 but v4 is due out soon based on the Illumos build. My testing was completed using v3.1.1.
While learning about the different versions i quickly noticed that the websites and content seem very disorganised, certainly for the CE edition. It took a while to realise that different domains represented different products, and even then the content is occasionally on the wrong site.
Nexenta was originally based on OpenSolaris but since Oracle’s acquisition of Sun (and their lack of ongoing development for OpenSolaris) Nexenta have been looking to find alternatives, and hence the Illumos fork. Another of Nexenta’s key selling points is its use of ZFS. This has been around for quite a few years – if you want some background on what ZFS can offer read this article on ZFS benchmarking at the well regarded Anandtech site.
It’s also worth mentioning that Tom Howarth has just joined Nexenta – maybe storage and VDI are a good combination? 😉
Installing NexentaStor on bare metal
Having decided to run Nexenta I fell at the first hurdle – finding the download (and I’m not the first). I went to the NexentaStor.org homepage and clicked the green Download button which took me to a page detailing my options – ISO, VM images etc. I wanted the VM images but scrolling down only revealed the .ISOs and there was no sign of the VM images. I then tried the ‘Downloads’ tab with no luck. Despite posting in the forums, I’ve still been unable to find them! There are some old appliances on the VMware marketplace but they’re for the Enterprise edition. I decided that downloading the .ISO was better than nothing although the download speed was very slow (10kbps) so I had to leave it running overnight (that could have been Internet or ISP related although it’s been consistently slow on the three occasions I’ve tried it).
Did I mention that the website seems at best disorganised? 🙁
With my download complete and a newly minted CD in hand I started my ‘bare metal’ install on my whitebox server. I assumed the install would be a ‘next’, ‘next’ process but unfortunately there were a few gotchas;
- Running the ‘memory burn-in test’ from the Grub bootloader failed with an error. This happened on both physical servers and a virtual server so I’m guessing it’s just broken.
- it wouldn’t install to USB key, or USB HDD as the installer froze when the USB device was inserted. This seems to be a known issue (despite being an acknowledged bug in 2008). Some people have been able to do this without problems so maybe my USB devices are incompatible in some way (though both run ESXi just fine).
- the installer froze for ages at the ‘Installing base appliance…’ stage which is something others have experienced too (see screenshot). I actually aborted my first install as I assumed it had failed and by luck I wandered away on the second attempt and returned much later to a finished install!
- during the install a serial number is generated which you need to register via nexenta.com (http://www.nexenta.com/corp/developer-edition-registration). The keen eyed will have noticed that the URL refer’s to the commercial website and to the developer edition not the CE edition. Did I mention that the website’s confusing?
Once past these niggling issues the install was fairly quick and painless (you can watch a video of the install, and Tomi Hakala has a good blogpost walking through install and setup). I also made some user errors which the support team were very quick to help sort out so kudos to them.
NOTE: It’s worth checking your hardware against the Nexenta HCL. There are some specific requirements if you’re using a VM (vt-d support).
Installing NexentaStor in a virtual machine
Next I tried installing NexentaStor as a VM because it offered a few advantages;
- more flexibility in how you carve up your hard disks. Nexenta requires you to dedicate a whole disk to volumes (or cache) but once you have a virtualisation layer you can easily work around that limitation. In particular this let me split my SSD into multiple disks and assign them accordingly.
- allows me to run other workloads on the same ESXi host
- ESXi will boot from USB leaving the internal HDD’s purely for VMs/NexentaStor
With a fresh install of vSphere5 up and running I then ran through the install process (you can follow guides here and here) using the same ISO image I used for the bare metal install. The process is essentially the same but there are a few key changes to be made afterwards;
- A single vCPU seems to work better compared to multiple vCPUs (see the comments on this article originally about installing VMtools in a Nexenta VM). Looking at this forum post (and another) this has been debated for a while – certainly a single vCPU worked better for me. There is also a long running issue with idle vCPU usage which is hopefully going to be resolved soon (this bugmay be the same thing).
- VMtools can be installed but it does need tweaking to get it working. Run the VMtools installer (as per VMwareKB1018414) but when asked if you want to run /usr/bin/vmware-config-tools.pl change the default [yes] to no. Using your favourite editor (vi or nano etc) search for ‘SUNW’ and then comment out the block which checks for the SUNWuiu8 package. Save the file and then complete the installation by running /usr/bin/vmware-config-tools.pl (full details in this article).
- There’s no support for the VMXNET3 vNIC (but plenty of discussion).
Even with the above tweaks I found performance was severely lacking. I posted on the Nexenta forums and also found this thread on the VMware communities about I/O issues in OpenSolaris 10 but found no resolution. After quite a bit of digging I found the napp-it guide to building a virtual ZFS NAS server which states;
‘A ZFS storage OS needs real hardware access to disk controller and attached disks for performance and failure handling reasons. We can virtualize the OS itself but we must allow exclusive pass-through access to a disk-controller and disks via vt-d.’
In vSphere parlance this means you need VMDirectPath to make the most of ZFS. Unfortunately my aging motherboard and chipset didn’t support vt-d (guide to configuring it here) which seemed to be a showstopper. Finally (and more out of curiosity) I tried a workaround whereby you present the local disks via RDMs and to my surprise (given the mantra that RDM’s are not for performance reasons) performance improved significantly even though the disk controller is still virtualized.
In the next article in the series I’ll cover my benchmarking results for both bare metal and virtual configurations along with some configuration issues. I’ll also cover my general thoughts about NexentaStor and whether it’s going to provide the storage for my home lab in the future…
Nexenta Planet – News and views on the Nexenta world (official Nexenta blog)
Thoughts from @VMstorage on Nexenta (plus Pivot3 and Diskeeper)
Twitterers you might want to follow if you’re interested in Nexenta;