www.vExperienced.co.uk

Firstly, this is not about the 1% associated with the Occupy WallSt campaign! As widely reported on Twitter and the blogosphere the 2012 vExpert program is up and running – I won’t go into the changes this year as there is plenty of coverage for that. In VMware’s own words;

The annual VMware vExpert title is given to individuals who have significantly contributed to the community of VMware users over the past year. The title is awarded to individuals (not employers) for their commitment to sharing their knowledge and passion for VMware technology above and beyond their job requirements.

Sounds great, let’s fill in that application form right? Before you apply have you ever paused to consider what is it you’re actually doing, and for whom? In an interesting article about ‘going social’ posted just a few weeks ago Dr Michael Hu talked about six myths companies believe are associated with a social strategy, one of which is the need to reach every customer to be effective. He refutes this, stating;

Instead, you need to discover the small number of “superfans” who want deeper engagement and then harness their enthusiasm to manage and strengthen other customer relationships on behalf of the brand. That’s the real power of community – you tend to the 1% who tend the other 99%.

That describes the vExpert in a nutshell - you are the 1%!

You could see this through cynical eyes as VMware using the community for their own benefit but like many of my peers I’ve been working in IT for well over a decade and virtualisation is the first time I’ve found a community that really benefits everyone involved. Maybe it’s the advent of social networking, maybe it’s the convergence of the various technologies or maybe it’s the time and effort expended by VMware (and geek herder extraordinaire @jtroyer)  but for some reason it works where it never did before. I enjoy being part of the VMware community and I  know it adds value for me (and therefore my employer) and many other people. While the 1% add great value on VMware’s behalf they also benefit greatly from the experience themselves. Just bear in mind that much as we’d all like VMware’s recognition, VMware need us too!

I’m already vExperienced and I’d love to be a vExpert. Fingers crossed!

ps. Apologies to Alex Maier who now runs the vExpert program – I’d already made up my ‘poster’ before I knew!

For the last few weeks we’ve been struggling with our vSphere5 upgrade. What I assumed would be a simple VUM orchestrated upgrade turned into a major pain, but I guess that’s why they say ‘never assume’!

Summary: there’s a bug in the upgrade process whereby NFS mounts are lost during the upgrade from vSphere4 to vSphere5;

• if you have NFS datastores with a space in the name
• and you’re using ESX classic (ESXi is not affected)

Our issue was that after the upgrade completed, the host would start back up but the NFS mounts would be missing. As we use NFS almost exclusively for our storage this was a showstopper. We quickly found that we could simply remount the NFS with no changes or reboots required so there was no obvious reason why the upgrade process didn’t remount them. With over fifty hosts to upgrade however the required manual intervention meant we couldn’ t automate the whole process (OK, PowerCLI would have done the trick but I didn’t feel inspired to code a solution) and we aren’t licenced for Host Profiles which would also have made life easier. Thus started the process of reproducing and narrowing down the problem.

• We tried both G6 and G7 blades as well as G6 rack mount servers (DL380s)
• We used interactive installs using a DVD of the VMware ESXi v5 image
• We used VUM to upgrade hosts using both the VMware ESXi v5 image and the HP ESXi v5 image
• We upgraded from ESXv4.0u1 to ESX 4.1 and then onto ESXiv5
• We used storage arrays with both Netapp ONTAP v7 and ONTAP v8 (to minimise the possibility of the storage array firmware being at fault)
• We upgraded hosts both joined to and isolated from from vCentre

Every scenario we tried produced the same issue. We also logged a call with VMware (SR 11130325012) and yesterday they finally reproduced and identified the issue as a space in the datastore name. As a workaround you can simply rename your datastores to remove the spaces, perform the upgrade, and then rename them back. Not ideal for us (we have over fifty NFS datastores on each host) but better than a kick in the teeth!

There will be a KB article released shortly so until then treat the above information with caution – no doubt VMware will confirm the technical details more accurately than I have done here. I’m amazed that no-one else has run into this six months after the general availability of vSphere5 – maybe NFS isn’t taking over the world as much as I’d hoped!  I’ll update this article when the KB is posted but in the meantime NFS users beware.

Sad I know, but it’s kinda nice to have discovered my own KB article. Who’d have thought that having too much space in my datastores would ever cause a problem?

In last night’s VMware Community podcast John Hall, VMware’s lead technical certification developer gave some tidbits of information about the upcoming VCAP5 exams;

• There will be an expedited path for those with VCAP4 certifications BUT they will be similar to the VCP upgrade in that it’ll be a time limited offer. He didn’t specify exactly what form this would take but with the VCP upgrade you have roughly six months to take the new exam with no course prerequisites.  I’m guessing you’ll have a similar period where the VCP5 prerequisite doesn’t apply.

• While not committing to dates he did state that exams might be available at the upcoming partner exchange which starts on Feb 13th 2012. Even if those dates slip expect them soon!

With the upcoming Feb 29th deadline for the VCP5 exam you’d better get your study skates on. If you don’t take the VCP5 before the 29th and you’re not in a position to take the the new VCAP5 exams in the ‘discount’ period (however long that turns out to be) you might find yourself needing to sit a What’s New course and passing the VCP5 exam before you’re even eligible for the VCAP5 exams. Not a pleasant thought!

UPDATE FEB 2012 – After some further testing I’ve concluded that this is a bigger pain than I previously thought. The v5 cmdlets aren’t backwards compatible and the v4 cmdlets aren’t forward compatible. This means that while you’re running a mixed environment with VMs on v4/v5 VMtools a single script can’t run against them all. Think audit scripts, AV update scripts etc. You’ll have to run the script twice, from two different workstations, one running PowerCLI v4 (against the v4 VMs) and one running PowerCLI v5 (against the v5 VMs). And I thought this was meant to be an improvement??

———- original article ————–

There are quite a few enhancements in PowerCLI v5 (there’s a good summary at Julian Wood’s site) but if you make use of the guest OS cmdlets proceed with caution!

We have an automated provisioning script which we use to build new virtual servers. This does everything from provisioning storage on our backend Netapps to creating the VM and customising configuration inside the guest OS. The guest OS configuration makes use of the ‘VMGuest’ family  of cmdlets;

• Invoke-VMScript
• Copy-VMGuestFile
• Get-VMGuest, Restart-VMGuest etc

Unfortunately since upgrading to vSphere5 and PowerCLI v5 we’ve discovered that the guest OS cmdlets are NOT backwards compatible! This means if you upgrade to PowerCLI v5 but your hosts aren’t running ESXiv5 and more importantly the VMTools aren’t the most up to date version any calls using the v5 cmdlets (such as Invoke-VMGuest) will no longer work. Presumably this is due to the integration of the VIX API into the base vSphere API – I’m guessing the new cmdlets (via the VMTools interface) now require the built-in API as a prerequisite.

As PowerCLI is a client side install the workaround is to have a separate install (on another PC for example) which still runs PowerCLI v4, but we have our vCenter server setup as a central scripting station (it’s simpler than every member of the team keeping up with releases, plugins etc) so this is definitely not ideal.

This is covered in VMware KB2010065.The PowerCLI v5 release notes are also worth a read.

Further Reading

Will Invoke-VMGuest work? (LucD)

Over the Christmas break I finally got some time to upgrade my home lab (link). One of my tasks was to build a new shared storage server and it was while installing the base ESXi (v5, build 469512) that I ran into an issue. I was unable to add any of the local disks to my ESXi host as VMFS datastores as I got the error “HostDatastoreSystem.QueryVmfsDatastoreCreateOptions” for object ‘ha-datastoresystem’ on ESXi….” as shown below;

The VI client error when adding a new datastore

I’d used this host and the same disks previously as an ESX4 host so I knew hardware incompatibility wasn’t an issue. Just in case I tried VMFS3 (instead of VMFS5) with the same result. I’ve run into a similar issue before with HP DL380G5′s where the workaround is to use the VI client connected directly to the host rather than vCentre. I connected directly to the host but got the same result. At this point I resorted to Google as I had a pretty specific error message. One of the first pages was this helpful blogpost at Eversity.nl (it’s always the Dutch isn’t it?) which confirmed it was an issue with pre-existing or incompatible information on the hard disks. There are various situations which might lead to pre-existing info on the disk;

• Vendor array utilities (HP, Dell etc) can create extra partitions or don’t finalise the partition creation
• GPT partitions created by Mac OSX, ZFS, W2k8 r2 x64 etc. Microsoft have a good explanation of GPT.

This made a lot of sense as I’d previously been trialling this host (with ZFS pools) as a NexentaStor CE storage server

Read more…

It’s that lovely time of year again (and I don’t mean Xmas!) when the next London VMware usergroup is open for registration! If you’re not familiar with the LonVMUG (where have you been?) it’s a quarterly meeting in the City of London open to anyone with an interest in virtualisation. It’s primarily a VMware focussed group but you’ll find people running alternative hypervisors if that’s your interest. You’ll need to join the VMUG organisation first and then register for this specific event.

If you haven’t attended before you may be wondering “What’s in it for me?”. Off the top of my head I’d say the following;

• Everyone at the LonVMUG has something to say and useful experiences. Find people with the same challenges as you and get talking!
• Hear about third party products (with demos)
• Get hands on with Labs
• Meet the experts and ask questions. There’s a lot of collective knowledge at the average VMUG with vExperts aplenty;
  • Fancy meeting one of a rare breed, a VCDX? Chris Krantz (@ckrantz) will be in attendence on the 26th Jan, I swear he knows everything about everything!
  • Into Powershell/PowerCLI? How about Jonathan Medd (@jonathanmedd) or Al Renouf (@alanrenouf) – Powershell gurus and book authors!
  • Using EMC at work or thinking of building a home or work lab? Seek out Simon Seagrave (@kiwi_si) – EMC and home labs guru
  • Maybe you’re an ISP and you want to know more about the VMware cloud offerings? Then seek out Simon Gallagher (@vinf_net)- vCloud specialist, vTardis inventor
  • Are you an SME with a broad interest in all things virtualisation? Barry Coombs (@virtualisedreal) is often along and specialises in this market.
  • Disaster recovery your thing? Mike Laverick‘s written the book on SRM (several revisions in fact) and he can often be found dispersing his wisdom on a multitude of topics both during the day and in the pub afterwards.
  • …and too many others to mention!
• Best of all this is all free!

The 26th Jan 2012 agenda (or download the PDF version);

It’s a great agenda and I’ll be supported a few friends who are presenting. Don’t miss out!

Where to go for the usergroup (make sure you register beforehand);

London Chamber of Commerce and Industry 33 Queen Street
London, EC4R 1AP (map)

Where to go for drinks afterwards;

The Pavilion End pub (official website)
23 Watling Street, Moorgate
London
EC4M 9BR (map)

Twitter:@lonvmug (or hashtag #lonvmug)

One of the most interesting sessions I attended at VMworld in Copenhagen was entitled ‘Cloud Computing 2012 to 2014 – a two year perspective’ (session CIM4603, subscription required). The speaker was Joe Baguley, a well known cloud evangelist who recently joined VMware as Chief Cloud Technologist. I’ve seen Joe present before at the Cloud Camp events so knew what to expect (humour, lots of snappy analogies and some thought provoking concepts) and I wasn’t disappointed (note the link above is to the same session from Las Vegas, presented with his own slant by David Hunter). If you’re interesed in hearing Joe’s speech in person I recommend registering for the national VMUG taking place on 3rd November in Birmingham.

One of Joe’s analogies (well quoted in the press) was to compare VM encapsulation to a shipping container. This isn’t anything new (Chuck Hollis explains it very well in this blogpost from 2008!) but it’s an analogy I’ve been thinking about since buying the book ‘The Box‘ for my wife as a Christmas present last year. As a commodity trader working with a team of shippers I thought she’d find a book about the history of the shipping container interesting (the New York Times listed it as one of the best business books ever written) but instead I found myself reading it during a weekend break. It didn’t take long to see parallels with what’s been happening over the last few years in the IT industry;

• Standardisation and automation altered existing business models – some companies flourished and others perished
• Whole professions changed and those who didn’t adapt found themselves out of work
• Containerisation introduces new challenges (scale, security)
• The container was used for many purposes beyond it’s original remit

In the four years since Chuck wrote his post the practice of cloud computing has advanced considerably. Whereas his focus (in that post at least) was networking it’s now clear that most areas of IT are being impacted from infrastructure to applications.

This isn’t a ‘technical how to’ blogpost with any conclusions but more of a ‘wandering thoughts, slow day at work’ post. I’m going to explore the analogy a bit further and include a few miscellaneous facts which were too good to ignore!

Read more…

Today was officially the start of VMworld Copenhagen even though many people were here yesterday for partner day. The hands on labs are always popular at VMworld shows, and for all the reasons previously covered by others. I’ve done two labs so far (HOL01, Creating the Hybrid Cloud and HOL27, Netapp and VMware) which were both useful in different ways. There’s a good atmosphere and the technology behind the labs continues to evolve – this year vCenter Operations (and I think Netapp Insight Balance) are on display showing how the lab infrastructure is performing. There are more seats and the labs are open longer than last year (32 hours) which is good to see.

I spent fair bit of time in the bloggers lounge, a small dedicated area with power, a separate wifi connection, and facilities for VMworld TV to broadcast live from. This is where you can often find John Troyer, the godfather of VMware’s social media scene along with many of the twittter names you’ve seen but never met in person. VMworld is a vertitable ‘who’s who’ of the virtualisation world – I found myself sitting next to Scott Lowe for ten minutes before realising who he was and saying hi! Many of the people hanging around the bloggers lounds have been at VMworld many times so it’s a good place to get a feel for what’s hot and what’s not at this year’s conference. I got my first taste of VMworld TV via an invite to vSoupTV. Quite a few people mentioned that it felt quieter this year but as the attendance has been confirmed at over 7,000 it must be because there’s more space rather than less people.

The centre of the complex is used as a relaxation zone complete with plenty of seating, food, recliners (for those quick power naps), table tennis, table ice hockey, chess sets etc. It’s a good place to meet people as you pass through on your way from a general session to the labs. Free wifi is available throughout the Bella Centre but unfortunately it’s pretty temperamental – somewhat expected for a large conference with over 7000 people. That wouldn’t be so bad but the VMworld iPhone app relies on internet access so when that’s not working you can’t reference your schedule or register for sessions. When it does work the VMworld iPhone app is pretty good – you can check for upcoming sessions, get a filtered twitter stream for a given session, and even check site maps. Read more…

As the complexity of virtual infrastructures increases it’s becoming harder to manage using conventional monitoring tools which were built with a more static environment in mind. In March 2011 VMware released the vCenter Operations product (vCOPS) to address this pain point. I’ve been running the 60 day trial at my company and now that the trial’s ending it’s time to share my thoughts.

What is vCOPS?

To quote the product page at VMware;

VMware vCenter Operations uses patented analytics and powerful visualizations to automate performance, capacity and configuration management. It collects and analyzes performance data, correlates abnormalities and identifies the root cause of building performance problems. VMware vCenter Operations provides capacity management to optimize resource usage and policy-based configuration management to assure compliance and eliminate sprawl and configuration drift. (emphasis my own)

The key differentiator is this promise to learn and understand the context of multiple metrics (CPU, memory, storage and network) and provide root cause analysis without you needing to manually define thresholds, benchmarks etc. Bear in mind that vCOPS is an infrastructure monitoring solution rather than application layer (which is more the domain on VMware’s AppSpeed, Quest’s solutions or ManageEngine’s Application Manager). I’m not the first blogger to cover this product so here’s some reading to get you up to speed;

• This great article by Bernd Harzog covers all editions of vCOPS along with some great analysis of it’s strengths and weaknesses.
• Eric Sloof’s blogpost describes how the product works in a bit more detail
• Kendrick Coleman’s post covers his own thoughts with some interesting comments.
• Chris Dearden has written an interesting post on monitoring including where he sees vCOPS fitting in.

While technically a ‘v1′ release the product comes from VMware’s purchase of Integrien (in August 2010) where it was originally marketed as VMAlive. Integrien have been working on the patented algorithms for several years so while the integration and VMware branding are new the guts of the product are not. VMware have published some YouTube videos or you can listen to VM Communities podcast #119 to get an overview of what vCOPS can offer. Read more…

Written by some of the top scripters in the VMware community the PowerCLI Reference book is really what it’s title states- a reference.  What it does (and does very well) is present both a ‘cookbook’ of useful scripts and explain how and why they work. While it does explain some concepts along the way it’s not really pitched as an introductory guide or as the best way to learn PowerCLI (Hal Rottenberg’s book might be better if this is what you’re after). The book is split into five main sections (see the full table of contents);

1. Install, configure and manage the vSphere environment. This section deals with vCenter automation, host deployment along with automated storage and networking provisioning.
2. Managing the VM lifecycle. Deals with creating, customising, and configuring VMs and vApps.
3. Securing vSphere. Covers backups, DR, security hardening and compliance.
4. Monitoring and reporting. Generating reports, statistical data, monitoring and auditing.
5. Scripting tools and features. Covers automation in general, the APIs (Get-View etc), Onyx, and common tools such as PowerGUI and PowerWF Studio. This chapter also covers adding a GUI to your scripts which is very useful for scripts that others need to use.

As you can see from the above list (and the fact it’s over 700 pages)  it covers a lot of material but despite this I’m impressed with the technical depth on each – I picked areas where my knowledge is strongest (though not in the same league as these guys) and still found myself learning something new everytime. For example I’ve used the VIX API while creating a scripted deployment for my test and dev environments at work and thought I knew it reasonably well.  To my surprise the book delved into the inner workings of the cmdlets themselves and explained how they in turn called some guest OS scripts which ship with PowerCLI. There was also had a good script for specifying a VM folder location via script, something I’d not implemented before as I couldn’t think of an easy way to specify the path. The index lists the pages where each cmdlet is used so it’s easy to look up the cmdlet you’re interested in and see code examples.

The scripts are downloadable from the book’s very own website and the authors have even put together a module containing all the code along with instructions for how to use it. This is a major bonus – you get nearly 80 prewritten functions you can integrate into your own scripts! These are useful for day to day administration, not just esoteric or niche functions. It’s worth checking this site out even if you’ve got the book – there are forums to discuss the scripts and at the moment they’re running a competition where to be in with a chance of winning you just have to take a photo of the book with a well known landmark in the background (ala ‘the orange HA book’ by Frank Denneman and Duncan Epping). I’m not sure how popular this will be as it’s a beast of a book to carry around, but that just means you’re chances of winning are that bit better!

It’s available in colour paperback or Kindle version (which is newly available again).

Disclosure – I’ve met both Jonathan Medd and Al Renouf at the VMware User Group on several occasions and was sent a copy of the book to review. There was no obligation to write a positive review and I’ve said it as I see it. I’d have bought the book anyway!

Further reading

• The recent London VMUG presentation on PowerCLI by Jonathan Medd
• The author’s websites: Al Renouf, Luk Dekens, Glenn Sizemore, Arnim van Lieshout, and Jonathan Medd with an honourable mention to Stu Radnidge as technical editor.