Home labs – a scalable vSphere whitebox
Having recently upgraded my home lab’s storage I decided it was also time to upgrade my aging hosts which date back to 2007. They’ve done well to survive and still be useful(ish) five years later but they’re maxed out at 8GB RAM and it’s becoming increasingly difficult to do anything with that. I briefly considered adding SSDs as host cache but that doesn’t address some of their other shortcomings such as no support for Fault Tolerance, VMDirectPath or any type of KVM functionality.
A quick look around the blogosphere revealed a few common options;
- The Baby Dragon II, (with a minor change via Chris Wahl)
- The HP Microserver range
- The HP ML110 G7 range
The problem for me was that these solutions all maxed out at 16 or 32GB RAM per host, a limitation of the single socket Xeon’s architecture. That’s a lot of memory for a home lab server today but to ensure that this server can last five years I really wanted more scalability. I wasn’t too fussed about noise as I use my cellar for my lab, and power consumption was a secondary concern. The server features of the Supermicro boards appeal to me (and many Supermicro motherboards are compatible with vSphere) so I browsed their range looking for the one that best met my requirements. My final parts list ended up as;
- CPU: 2 x Intel Xeon E5-2603, 1.8GHz, 4 cores, 80W (£157 each at Lambdatek)
- Heatsink: Supermicro SNK-P0048AP4 (£28 at Lambdatek)
- Motherboard: Supermico X9DRL-3F, dual CPU socket, up to 256GB RAM, onboard SAS & SATA, IPMI/KVM, dual Gb NICs (£330 at Lambdatek)
- RAM: 64GB (8*8GB) Samsung 240 pin DDR3 non-ECC UDIMM, 10600 PC3 (£300 at scan.co.uk)
- Case: Lian-Li PC-A05NB (£65 at Scan.co.uk)
- Power supply: Silverstone Strider Plus 600W (£85 at Scan.co.uk)
The total cost comes to around £1150. I’m branding mine the vHydra after the mythical multi-headed dragon!
Note: In the US this is significantly cheaper, coming in at $1450, or about £900.
For the money I get a powerful server that can replace all three of my current 8GB hosts and more than match their performance while consuming less power and space, plus compatibility with all VMware features including FT (and full EPT for nested ESXi). With two onboard storage controllers, support for SAS drives, and VMDirectPath I can run a Nexenta VM to supplement my shared storage plus I gain remote KVM and the option of future memory upgrades (up to a max of 256GB RAM!). If you want to save some cash you can always buy a single CPU (and fewer DIMMs) and add the second later if your requirements increase.
NOTE: FT is supported on the Xeon E5-26xx CPUs but at the time of writing VMware’s SiteSurvey tool hasn’t been updated so will erroneously report it’s not supported. Given that the web client is the way forward it’s quite possible SiteSurvey won’t be updated in its current VI client format.
It might sound crazy expensive but it’s less than an extra level MacBookPro with anti-glare screen which makes it sound more reasonable (to me at least). If it helps my career over four years it seems like a good investment. Compared to a single Baby Dragon II it’s more expensive and larger (ATX vs micro-ATX) but if you’re considering two Baby Dragons it’s a closer call – it costs slightly less, has less compute (no hyperthreading and a lower core MHz) but more memory scalability (although you’ll need RDIMMs), SAS support, and a slightly smaller footprint. With two Baby Dragons however you have more flexibility as you have two physical instances of ESXi – no nesting and a ‘proper’ cluster is quite compelling. Two servers is also better for experimenting with other technologies (hypervisors which may not work nested) etc but I went with my ‘vHydra’ because I had a case, power supply, and some RDIMM’s which I could reuse (which the Baby Dragon wouldn’t accept) – your circumstances may vary.
Tip: It’s worth bearing in mind the 32GB limit on the free version of ESXi – unless you’re a vExpert or they reinstate the VMTN subscription you’ll be stuck with 60 day eval editions if you go above 32GB (or buying a licence!). For a full discussion of scale up vs scale out I’d recommend the VCAP-DCD Design Workshop. Having considered scale up vs out for my lab I think I’ve got enough thoughts for a future blogpost…
|Baby dragon II||The vHydra|
|Compute||4 cores @ 3.2GHz|
|8 cores @ 1.8GHz
(Max 256GB using RDIMMs
Max 64GB using UDIMMs)
One 82579LM (custom driver)
9.6" x 9.6"
12" x 10"
|Case size||279mm x 262mm x 373mm|
(W x D x H)
|210mm x 381mm x 490mm
(W x D x H)
|Power||80W||190W (see below for update)|
Motherboard and CPU choices
The X9DRL-3F motherboard is relatively new having been launched in March this year. Being dual socket it requires Xeon E5-26xx series processors (tech details here, reviewed here, with some performance figures via El Reg) which are also new having been launched in May this year. The Xeon E5 CPU range, like the Xeon 5xxx range previously, is designed for high end server and workstations and crucially for me supports much higher memory density;
- The cheapest is the E5-2603 (£157 at Lambdatek) with a clock speed of 1.8GHz and four cores, although no hyperthreading. The max memory speed is 1066Mhz whereas the motherboard will support 1333MHz.
- Next up is the E5-2609 (£230 at Lambdatek) with a clock speed of 2.4GHz, four cores, and again no hyperthreading and a max memory speed of 1066MHz.
- Another option is the E5-2620 (£305 at Lambdatek) with a clock speed of 2.0GHz and six cores plus hyperthreading, giving 12 threads for VMs plus it supports 1333MHz memory.
The Xeon E5-26xx range is comprehensive so if you need more compute power you can easily scale up if you have the funds.
Tip: You’ll need to order a heatsink separately as it doesn’t come with the CPU even if you buy the retail version. Details of the Intel heatsinks are in the above review link and the Supermicro ones can be found in this PDF (I went with the SNK-P0048AP4 which is fine for a standard ATX case). Be careful if you buy a different motherboard/CPU combination as some Supermicro boards use an uncommon ‘narrow ILM’ socket fixing. This board uses the more common square ILM. The two CPU sockets are quite close together so some of the aftermarkets heatsinks might not fit.
The motherboard has 8 DIMM slots but you need both CPUs to use all eight slots – with a single CPU the most you can use is four. It’s flexible as it’ll accept both buffered/registered and unbuffered RAM, although you’ll need to use registered memory if you want to go above 64GB.
Tip: In the UK most resellers don’t hold stock of the Supermicro motherboards and there’s a two week wait while they’re ordered from the US. Frustrating!
Power supply choices
Being a dual socket board you’ll need a bit more power compared to a desktop. You need a PSU that’s compliant with ATX 2.02 and SSI standards (often referred to as EPS12V) which means it’ll have the usual 24 pin connector and at least one 8 pin (or a 4+4pin) CPU connector. This motherboard needs a PSU with two eight pin power connectors which is less common – I had two EPS12V certified PSUs and neither had enough. I went for the 600W Silverstone Strider Plus as it’s their lowest spec PSU which offers dual EPS12V connectors and it gets good reviews. If you go for a different PSU make sure it’s not too deep – one of my PSUs was 180mm (vs 160mm for the Silverstone) and it wouldn’t fit the Lian-Li case without hitting the first CPUs RAM modules. I used Thermaltake’s PSU calculator to size my PSU correctly.
UPDATE – 24th Oct 2012 – if you get voltage warnings on the second CPU socket try updating your IPMI firmware.
UPDATE – 26th Nov 2012 – I’ve been monitoring power usage and my vHydra idles (with a few VMs that do nearly nothing) at around 85W. With electricity prices around 12.8p/kWh that’s approx 25p/day, or £1.75/wk, or £91/yr for this one server so factor that into your lab costs.
Building a nested lab (YellowBricks)
vTardis lightening talk at VMworld Barcelona (Simon Gallagher)
VMware vCloud “In A Box” for Your Home Lab (Chris Colotti)
A whitebox is for sissies! (Eric Sloof)