www.vExperienced.co.uk

IT is a fast moving industry, and the current shift to cloud computing is accelerating the rate of change. With the forthcoming release of the vSphere5 VCAP exams (VCAP5-DCA and the VCAP5-DCD) I’ve been planning to study my socks off to get both exams under my belt. I’m sure I’m not alone in struggling to make study time, particularly as my day job doesn’t tend to cover all of the material either because we’re only Enterprise level licensing or because we don’t use all the features. On top of that I’d like to update my VCAP-DCA study guide which will consume a significant chunk of time.

This serious case of study contention has made me revisit my priorities. The VMware ecosystem has evolved considerably over the last few years and there’s considerable buzz around VMware View and vCloud Director (not to mention the whole Cloud Foundry and vFabric ecosystem). Maybe these would be better areas to focus on? As John Troyer said about certifications “You don’t have to collect them all!” but it does make me wonder – which VMware certs are most likely to benefit career progression?

VCP or VCAP?

First some facts. The VMware platform which evolved into vSphere was launched in 2001, the first VCP exam was available in June 2003 (@susangude is VCP#1), and the first VCAP exams were launched seven years later in August 2010. Nine years on and the statistics are well known – around 60,000 VCPs worldwide and approximately 600 certified in each of the two VCAP tracks (so 50 VCPs for every VCAP!).

Read more…

After a course postponement in January (due to lack of attendees) I finally took the VMware Design Workshop (v5) this week, a three day course designed to help you prepare for the VCAP-DCD exam as well as real life solution design using VMware vSphere. If you’ve sat the v4 course there’s very little difference so you can probably stop reading now…

I’ve been responsible for the same vSphere implementation at my company for well over four years now, so unlike contractors or consultants I’m not seeing new infrastructures every week and I’m not redesigning mine week to week either. So why go on the workshop you ask? I hoped that spending some time away from the office thinking about design might allow me a fresh look at how we currently have our environment configured as well as giving me a gauge on my own skillset. There was a select group of four on the course which was a slight shame as I think more people would have increased discussions and added value (everyone does things differently and has different circumstances). Despite the limited numbers we had diverse opinions and experience covering government, army, SME and enterprise environments. Luckily our instructor Paul McSharry (@pmcsharry) is one of those trainers who’s also still consulting/contracting so he was more than able to fill any gaps with real world experience including some great discussions around VMware’s vCloud in particular (although neither vCD nor vCloud are on the blueprint they made for great examples). I’ve was lucky to have Mike Laverick for my ICM course back in 2007 and equally lucky to have Paul, he’s a great trainer.

The workshop is still a three day course with minimal hands on work – it’s all whiteboards and discussions. That in itself is quite refreshing as many courses are ‘heads down’ in a PC racing through labs whereas this course is more social. It also meant I kept my mind off work for the three days as I didn’t have a PC in front of me as a distraction! It looks as if the order of the modules has changed from the v4 course but the content is largely the same;

• Day one – Course introduction, the design process, and storage design
• Day two – Network design, host (compute) design, and virtual machine design
• Day three – Virtual datacenter design, management and monitoring design

I was surprised to see some topics which I expected to be ‘bread and butter’ subjects get minor coverage;

• The section on storage design didn’t cover RAID to any degree and simply states ‘For the majority of VM workloads the RAID level does not matter for performance if the array has sufficient battery-backed cache’ along with ‘Select RAID level based on availability requirements’. As a Netapp guy I agree it generally doesn’t matter – large numbers of spindles are put into aggregates anyway rather than the older RAID group per LUN philosophy of low end MSA arrays etc. The lab exercises didn’t include any IOps, latency or throughput figures either but as Paul rightly pointed out that could overcomplicate the design process given the time constraints, plus it’s a vSphere design course not a specialised storage design.
• The host design section did cover cluster scale up vs scale out but briefly (given the column inches on blogposts!). Without going into vendor specific detail this topic is rather tricky as many companies will have standards to adhere to, preferred choices (personal or corporate) and extra constraints. That’s exactly what the course states – organisational constraints are most likely what you’ll have to contend with.
• There is a single page on licensing which simply states ‘buy the license version that supports the features required in your design’. Thanks for that, great way to gloss over the controversial vRAM!
• I also expected a heavy focus on the new features of v5 (Autodeploy, Storage DRS etc) but while they were covered it wasn’t in depth. On reflection that’s probably correct as there are so many things to consider for the average design that the basics are still the same.

Every vendor course has some ‘best practices’ which you take with a pinch of salt and this course is no different. ‘Always use jumbo frames’, ‘Always buy the fastest CPU you can afford’, ‘Prefer distributed switches’. Keep your questioning hat on!

The lab scenarios didn’t always give enough information but that does at least make them flexible – there’s no reason why you can’t set yourself some constraints. There was no requirement around storage protocol choice for example so I found myself picking NFS but having to justify it based on other elements of the design. Cost wasn’t mentioned in mine beyond ‘it should be cost effective’ so you have to make you own mind up – would vCSHB be worthwhile for example? I decided not and as long as you understand the implications of your choices you’ve achieved your goal.

Overall I found the course very useful although given that I’m not doing design as part of my job I suspect I’ll find the exam very tough. The blogosphere covers a lot of relevant material which the course skims over due to time restrictions – I doubt the workshop alone would be sufficient to pass the exam. Now I just have to hit the whitepapers and the design bible (VMware vSphere design by Forbes Guthrie, Scott Lowe, and Maish Saidel-Keesing) while I wait for the v5 exam to be officially released. Given that the beta has already come and gone I don’t think I’ll have long to wait!

UPDATE May 2012 – The exam has now been released. You can get details on the official VMware Education page and sign up for authorisation here.

Further Reading

Sean Crookston has a collection of links from other workshop reviews.

The VCAP-DCD forums

The VCP5 exam is;

A. Remarkably easy, a bunch of monkeys typing randomly would probably pass.

B. Unreasonably hard, even a team of VCDX’s working together would be lucky to scrape a pass.

C. A fair test of real world knowledge. If you’ve worked with vSphere for a while and v5 in particular you’ll be fine.

D. all about Veuve Clicqout Ponsardin, a premium brand of champagne.

Choose one.

Hint – one answer is related, but not correct!

———————————————–

UPDATE March 2012 – VMware have just confirmed that the fix will be released as part of vSphere5 U2. Interesting because as of today (March 15th) update 1 hasn’t even been released – how much longer will that be I wonder? I’m also still waiting for a KB article but it’s taking it’s time…

UPDATE May 2012 – VMware have just released article KB2013844 which acknowledges the problem – the fix (until update 2 arrives) is to rename your datastores. Gee, useful… 

———————————————–

For the last few weeks we’ve been struggling with our vSphere5 upgrade. What I assumed would be a simple VUM orchestrated upgrade turned into a major pain, but I guess that’s why they say ‘never assume’!

Summary: there’s a bug in the upgrade process whereby NFS mounts are lost during the upgrade from vSphere4 to vSphere5;

• if you have NFS datastores with a space in the name
• and you’re using ESX classic (ESXi is not affected)

Our issue was that after the upgrade completed, the host would start back up but the NFS mounts would be missing. As we use NFS almost exclusively for our storage this was a showstopper. We quickly found that we could simply remount the NFS with no changes or reboots required so there was no obvious reason why the upgrade process didn’t remount them. With over fifty hosts to upgrade however the required manual intervention meant we couldn’ t automate the whole process (OK, PowerCLI would have done the trick but I didn’t feel inspired to code a solution) and we aren’t licenced for Host Profiles which would also have made life easier. Thus started the process of reproducing and narrowing down the problem.

• We tried both G6 and G7 blades as well as G6 rack mount servers (DL380s)
• We used interactive installs using a DVD of the VMware ESXi v5 image
• We used VUM to upgrade hosts using both the VMware ESXi v5 image and the HP ESXi v5 image
• We upgraded from ESXv4.0u1 to ESX 4.1 and then onto ESXiv5
• We used storage arrays with both Netapp ONTAP v7 and ONTAP v8 (to minimise the possibility of the storage array firmware being at fault)
• We upgraded hosts both joined to and isolated from from vCentre

Every scenario we tried produced the same issue. We also logged a call with VMware (SR 11130325012) and yesterday they finally reproduced and identified the issue as a space in the datastore name. As a workaround you can simply rename your datastores to remove the spaces, perform the upgrade, and then rename them back. Not ideal for us (we have over fifty NFS datastores on each host) but better than a kick in the teeth!

There will be a KB article released shortly so until then treat the above information with caution – no doubt VMware will confirm the technical details more accurately than I have done here. I’m amazed that no-one else has run into this six months after the general availability of vSphere5 – maybe NFS isn’t taking over the world as much as I’d hoped!  I’ll update this article when the KB is posted but in the meantime NFS users beware.

Sad I know, but it’s kinda nice to have discovered my own KB article. Who’d have thought that having too much space in my datastores would ever cause a problem?

Over the Christmas break I finally got some time to upgrade my home lab. One of my tasks was to build a new shared storage server and it was while installing the base ESXi (v5, build 469512) that I ran into an issue. I was unable to add any of the local disks to my ESXi host as VMFS datastores as I got the error “HostDatastoreSystem.QueryVmfsDatastoreCreateOptions” for object ‘ha-datastoresystem’ on ESXi….” as shown below;

The VI client error when adding a new datastore

I’d used this host and the same disks previously as an ESX4 host so I knew hardware incompatibility wasn’t an issue. Just in case I tried VMFS3 (instead of VMFS5) with the same result. I’ve run into a similar issue before with HP DL380G5′s where the workaround is to use the VI client connected directly to the host rather than vCentre. I connected directly to the host but got the same result. At this point I resorted to Google as I had a pretty specific error message. One of the first pages was this helpful blogpost at Eversity.nl (it’s always the Dutch isn’t it?) which confirmed it was an issue with pre-existing or incompatible information on the hard disks. There are various situations which might lead to pre-existing info on the disk;

• Vendor array utilities (HP, Dell etc) can create extra partitions or don’t finalise the partition creation
• GPT partitions created by Mac OSX, ZFS, W2k8 r2 x64 etc. Microsoft have a good explanation of GPT.

This made a lot of sense as I’d previously been trialling this host (with ZFS pools) as a NexentaStor CE storage server

Read more…

In my article The Good, the Bad, and the Ugly I discussed the controversial licencing change which is coming with vSphere5. Many people are saying they’ll move to a competing hypervisor to escape these potentially higher license fees and even though my company aren’t facing this issue (our vRAM entitlement is sufficient in the short term at least) at some point my management team are going to (or should!) ask me to justify the expense and whether there are suitable alternatives. Most people I speak to acknowledge that the competition can’t compare with vSphere for features or maturity but they do discuss when they’ll be ‘good enough’ to satisfy the more basic requirements (and at a cheaper price?). So is now the time for the competition to shootdown vSphere?

‘Gunfight at the ‘OK?’ corral!

I needed facts so I set out to see how feasible a change would be and if the benefits were justified. For the purposes of this article I’m going to concentrate on the three main virtualisation vendors recognised as leaders by Gartner – VMware (vSphere), Citrix(XenServer) and Microsoft (Hyper-V). I’m also going to focus purely on my own environment – I don’t know XenServer or Hyper-V well enough to do a general purpose comparison and there are too many factors to consider in a single blogpost.
PS. If you’re after a general comparison  I’d suggest starting with Andreas Groth’s virtualisation matrix. This excellent site lets you see at a glance the feature sets of the three main hypervisors and even generate custom reports. Note that the site starts with the free version of ESXi and XenServer selected for comparison. You can use the menus on the left to change the version for each solution etc as required – nice!

Before even worrying about general performance, stability, quality of support, roadmaps etc I thought I’d do a feature check specific to my environment. We’re primarily using our VMware platform for server consolidation – we’ve done the P2V game for all but a few tier1 apps and now use it heavily for dev and test environments which are 100% virtual. As an Enterprise (not Enterprise+) licencee we don’t have access to some of the higher end features (distributed switches, host profiles, SIOC) nor are we using the extended VMware ecosystem such as SRM, Cloud Director, Orchestrator etc. Given our relatively simple use of virtualisation I suspected we’d be a good candidate for the ‘good enough’ competitors.  Comparing vSphere Enterprise vs Hyper-V Enterprise vs XenSever Enterprise Edition I found that;

• We use storage vMotion all the time to rearrange our underlying storage for capacity or performance reasons, or to migrate to new Netapp arrays etc. Moving to a rival hypervisor would mean losing this functionality as neither XenServer of Hyper-V offer a completely nondisruptive migration. Given the downtime this would cause the business it would either result in lots of out of hours work (with associated overtime costs) or disruption to the business – both of which I know they’d rather pay more to avoid.
• Alongside various flavours of Windows we run a significant number of Oracle Enterprise Linux  and Red Hat Enterprise Linux servers. When I last looked back in early 2010 Hyper-V only supported a single vCPU for Linux VMs and while it now supports vSMP (up to 4, same as our Enterprise licence of vSphere) only RHEL and SUSE are officially supported. A quick Google shows that OEL does work but that’s another argument altogether. Xenserver supports both RHEL and Oracle Enterprise Linux (v4 and v5, both of which we use).
  
• We use plenty of VLANs on our ESX blades (HP C class) which Hyper-V would work with but XenServer would not. It requires management ports to be ‘access ports’ and in blades with limited pNICs we’d have a problem. We could work around it using HPs Virtual Connect, Xsigo etc but that’s more cost and complexity.
• We currently use NFS for the majority of our VMware estate and while our underlying storage arrays offer both FC and iSCSI (and we have a SAN fabric in place) it’s not a change we’d make lightly. XenServer supports NFS but Hyper-V does not. We have inhouse expertise on other protocols but it means changing our processes, provisioning scripts, documentation, training etc. It’s also a significant technical change so would consume quite a lot of time in change requests and implementation. Management would want to clearly justify the time and risks involved.
• We currently get nearly 50% memory overcommit on our ESX hosts, a feature which saves us money on hardware purchases and isn’t available in either competing hypervisor. Hyper-V does offer Dynamic Memory but it doesn’t work with Linux VMs, which rules it out for us. With vSphere5 and the new vRAM licensing this benefit is largely lost however.
• We’ve used Update Manager to a significant degree and while Hyper-V offers similar functionality via WSUS (which we already have deployed), XenServer is more limited.

Conclusions

For my specific circumstances the competition is not ‘OK’ because we’d lose functionality we rely on.

This will vary for everyone and will be completely different if you’re just starting down the virtualisation road and don’t have a feature-set to match up to (in which case this VMware vs XenServer cost calculator or VMware vs Hyper-V cost comparison might help). Could we work around all the issues above? Sure we could, but would it be cost effective? Having already paid for our VMware licensing we aren’t going to simply drop the technology however, at best we’d add new capacity using an alternative hypervisor and slowly migrate all hosts to the new platform. If we did go down that road then we’d have the challenge of running a multi-hypervisor infrastructure at least in the short term – increased training, increased complexity, limited toolsets (most support a single hypervisor only), interoperability issues etc.

The whole reason behind this research was to see if we could save money, and if that in turn justified a switch. This is always tricky as it’s rarely an ‘apples to apples’ comparison but my brief findings were that any cost saving would be eaten up by new toolsets, training, migration costs etc. I’d also note that as we’re entitled to vSphere5′s new features for no extra cost the competition is going to have to improve futher still to make this change feasible in the future.

If the recent licensing change means your costs will increase or you just want to reduce vendor lock in I’d recommend doing the same comparison for your infrastructure to see how feasible a change really is. I suspect VMware are able to raise prices (even if only for the alleged minority) because they know that for most people it’s not a viable or particularly attractive option.

Further reading

Is Hyper-V good enough?

This free online training from Microsoft Virtual Academy is a good place to learn more about Hyper-V.

Xenserver and Hyper-V make the ‘leaders’ quadrant

Why VMware continues to dominate despite Hyper-V advances

The announcement on 12th July about vSphere5 was largely overshadowed by the furore around licensing changes. My gut reaction was much like many people – angry that VMware seemed to be charging more for the same functionality. If you want a feel for customer feedback, this VMware communities thread is a good place to start or see how many posts on the ESXi v5 forums relate to licensing. I’ve now reached phase 5 of ‘the LonelySysAdmin’s 5 stages of VMware licensing grief‘ – acceptance.

The Good

• I’ve done the maths for my environment (thanks to Hugo Peters for the PowerCLI script to check) and I’m one of the 90% that VMware claim will see no increase in costs. We’re using about 62% of our vRAM entitlement (using 2.1TB from 3.4TB allowable) so have some growth factored in. So far, so good and not a big surprise as I knew we didn’t push our current infrastructure too hard.
• At the recent London VM user group there was a similar feeling – many people were OK with the licensing today but had concerns about the future.
• There are no longer any restrictions on number of cores per socket. My company use Enterprise rather than Enterprise+ so without this change we’d be restricted to six cores per socket, a limit we’ve already reached.
• Service providers aren’t affected by the recent changes. They’re already on a different licensing model which isn’t based on vRAM (the VMware Service Provider Program)
• New VDI users can use the vSphere Desktop edition which doesn’t include the vRAM based license model. Our company haven’t gone down the VDI route yet, so we’re not impacted by the upgrade issues (see below).

Read more…