———————————————–
UPDATE March 2012 – VMware have just confirmed that the fix will be released as part of vSphere5 U2. Interesting because as of today (March 15th) update 1 hasn’t even been released – how much longer will that be I wonder? I’m also still waiting for a KB article but it’s taking it’s time…
UPDATE May 2012 – VMware have just released article KB2013844 which acknowledges the problem – the fix (until update 2 arrives) is to rename your datastores. Gee, useful… 
———————————————–
For the last few weeks we’ve been struggling with our vSphere5 upgrade. What I assumed would be a simple VUM orchestrated upgrade turned into a major pain, but I guess that’s why they say ‘never assume’!
Summary: there’s a bug in the upgrade process whereby NFS mounts are lost during the upgrade from vSphere4 to vSphere5;
- if you have NFS datastores with a space in the name
- and you’re using ESX classic (ESXi is not affected)
Our issue was that after the upgrade completed, the host would start back up but the NFS mounts would be missing. As we use NFS almost exclusively for our storage this was a showstopper. We quickly found that we could simply remount the NFS with no changes or reboots required so there was no obvious reason why the upgrade process didn’t remount them. With over fifty hosts to upgrade however the required manual intervention meant we couldn’ t automate the whole process (OK, PowerCLI would have done the trick but I didn’t feel inspired to code a solution) and we aren’t licenced for Host Profiles which would also have made life easier. Thus started the process of reproducing and narrowing down the problem.
- We tried both G6 and G7 blades as well as G6 rack mount servers (DL380s)
- We used interactive installs using a DVD of the VMware ESXi v5 image
- We used VUM to upgrade hosts using both the VMware ESXi v5 image and the HP ESXi v5 image
- We upgraded from ESXv4.0u1 to ESX 4.1 and then onto ESXiv5
- We used storage arrays with both Netapp ONTAP v7 and ONTAP v8 (to minimise the possibility of the storage array firmware being at fault)
- We upgraded hosts both joined to and isolated from from vCentre
Every scenario we tried produced the same issue. We also logged a call with VMware (SR 11130325012) and yesterday they finally reproduced and identified the issue as a space in the datastore name. As a workaround you can simply rename your datastores to remove the spaces, perform the upgrade, and then rename them back. Not ideal for us (we have over fifty NFS datastores on each host) but better than a kick in the teeth!
There will be a KB article released shortly so until then treat the above information with caution – no doubt VMware will confirm the technical details more accurately than I have done here. I’m amazed that no-one else has run into this six months after the general availability of vSphere5 – maybe NFS isn’t taking over the world as much as I’d hoped! I’ll update this article when the KB is posted but in the meantime NFS users beware.
Sad I know, but it’s kinda nice to have discovered my own KB article. Who’d have thought that having too much space in my datastores would ever cause a problem?
Host Profiles are a new feature to vSphere 4 but are only available to Enterprise+ licencees. As my company haven’t yet found a need for Enterprise+ features I’d not really worked with them before so this section was new to me. Interestingly the main reference given in the blueprint is the Datacenter Administration Guide which has very little about host profiles. The ESX/ESXi configuration guides have a small section on host profiles but not much, so the best reference is probably the VMware Host Profiles – Technical Overview whitepaper.
Skills and Abilities
- Use Profile Editor to edit and/or disable policies
- Create sub‐profiles
- Use Host Profiles to deploy vDS
Tools & learning resources
Host Profiles (VCP revision)
Basically host profiles are the equivalent of Microsoft’s Group Policy, but for VMware hosts.
- Two primary uses
- Ease deployment challenges (faster, more consistent)
- Ongoing configuration control and audit reporting
- Policy options (determining how a configuration setting is applied)
- Use a fixed configuration
- Ask the user how to configure it
- Use an intelligent policy (using one or multiple criterion)
- Disregard a setting
- Works in a similar fashion to Update Manager;
- Create a baseline from a reference host.
- Attach the host profile to the hosts or clusters you want to configure
- Remediate (configure) the hosts or clusters
- Review compliance status
- Unlike VUM it can’t remediate all the hosts in a cluster automatically (it won’t put them into maintenance mode for you etc). You can attach a profile to the cluster but you have to apply to each host manually (this is largely because the host profile may require user input).
- Can only be used on vSphere hosts (not VI 3.x)
- Must be created using a reference host, or imported from a previously created host profile.
- Can be exported (in VMware Profile Format, *.vcf, which is XML content). Host Profiles are not shared using vCentre Linked Mode, you have to export/import to other vCentre instances.
NOTE: Administrator passwords aren’t exported as a security measure.
- An ESX reference host can be applied to either ESX or ESXi. An ESXi reference host can ONLY be applied to another ESXi host.
- When updating a host using a host profile you have to manually put the host in maintenance mode first. This is a significant issue for some people (although if you’re licenced for host profiles you’ve also got licences for vMotion and DRS so moving VMs off the host is potentially easier). Note that you need to enter maintenance mode even for trivial settings such as setting the time, timezone etc. Any setting which normally requires a reboot (changing service console memory for example) will still need a reboot.
- You must have both host profile privileges (create, delete, edit etc) AND privileges to configure the area in question (Networking, Storage etc) for the operation to be allowed.
REAL WORLD: When building a new ESX/ESXi host it will have a 60 day eval period with all features enabled so even if you don’t have Enterprise+ licencing you can use host profiles for initial configuration.
Read more…
Knowledge
- Identify resxtop/esxtop metrics related to memory and CPU
- Identify vCenter Server Performance Chart metrics related to memory and CPU
Skills and Abilities
- Troubleshoot ESX/ESXi Host and Virtual Machine CPU performance issues using appropriate metrics
- Troubleshoot ESX/ESXi Host and Virtual Machine memory performance issues using appropriate metrics
- Use Hot‐Add functionality to resolve identified Virtual Machine CPU and memory performance issues
Tools & learning resources
This is another objective that’s hard to quantify – experience will be the main requirement! There are some great general purpose resources out there;
Note that resxtop (built in to the vMA) does not offer the ‘replay’ mode available in ESX classic. Source: VMworld session MA6580, vMA Tips and Tricks. Read more…
Knowledge
- Identify virtual switch entries in a Virtual Machine’s configuration file
- Identify virtual switch entries in the ESX/ESXi Host configuration file
- Identify CLI commands and tools used to troubleshoot vSphere networking configurations
- Identify logs used to troubleshoot network issues
Skills and Abilities
- Utilize net-dvs to troubleshoot vNetwork Distributed Switch configurations
- Utilize vicfg-* commands to troubleshoot ESX/ESXi network configurations
- Configure a network packet analyzer in a vSphere environment
- Troubleshoot Private VLANs
- Troubleshoot Service Console and vmkernel network configuration issues
- Troubleshooting related issues
- Use esxtop/resxtop to identify network performance problems
- Use CDP and/or network hints to identify connectivity issues
- Analyze troubleshooting data to determine if the root cause for a given network problem originates in the physical infrastructure or vSphere environment
Tools & learning resources
Identify virtual switch entries in a VMs configuration file
Contains both vSS and vDS entries;
In the example VM below it has three vNICs on two separate vDSs. When troubleshooting you may need to coordinate the values here with the net-dvs output on the host;
- NetworkName will show “” when on a vDS.
- The .VMX will show the dvPortID, dvPortGroupID and port.connectid used by the VM – all three values can be matched against the net-dvs output and used to check the port configuration details – load balancing, VLAN, packet statistics, security etc
NOTE: Entries are not grouped together in the .VMX file so check the whole file to ensure you see all relevant entries.
Identify virtual switch entries in the ESX/i host configuration file
The host configuration file (same file for both ESX and ESXi);
Like the .VMX file it contains entries for both switch types although there are only minimal entries for the vDS. Most vDS configuration is held in a separate database and can be viewed using net-dvs (see section 6.3.7).
Command line tools for network troubleshooting
The usual suspects;
- vicfg-nics
- vicfg-vmknic
- vicfg-vswitch (-b) for CDP
- vicfg-vswif
- vicfg-route
- cat /etc/resolv.conf, /etc/hosts
- net-dvs
- ping and vmkping
This section overlaps with objectives 1.1 (Advanced storage management) and 1.2 (Storage capacity) but covers the multipathing functionality in more detail.
Knowledge
- Explain the Pluggable Storage Architecture (PSA) layout
Skills and Abilities
- Install and Configure PSA plug‐ins
- Understand different multipathing policy functionalities
- Perform command line configuration of multipathing options
- Change a multipath policy
- Configure Software iSCSI port binding
Tools & learning resources
- Product Documentation
- vSphere Client
- vSphere CLI
- VMware KB articles
Understanding the PSA layout
The PSA layout is well documented here, here. The PSA architecture is for block level protocols (FC and iSCSI) – it isn’t used for NFS.
Terminology;
- MPP = one or more SATP + one or more PSP
- NMP = native multipathing plugin
- SATP = traffic cop
- PSP = driver
There are four possible pathing policies;
- MRU = Most Recently Used. Typically used with active/passive (low end) arrays.
- Fixed = The path is fixed, with a ‘preferred path’. On failover the alternative paths are used, but when the original path is restored it again becomes the active path.
- Fixed_AP = new to vSphere 4.1. This enhances the ‘Fixed’ pathing policy to make it applicable to active/passive arrays and ALUA capable arrays. If no user preferred path is set it will use its knowledge of optimised paths to set preferred paths.
- RR = Round Robin
One way to think of ALUA is as a form of ‘auto negotiate’. The array communicates with the ESX host and lets it know the available path to use for each LUN, and in particular which is optimal. ALUA tends to be offered on midrange arrays which are typically asymmetric active/active rather than symmetric active/active (which tend to be even more expensive). Determining whether an array is ‘true’ active/active is not as simple as you might think! Read Frank Denneman’s excellent blogpost on the subject. Our Netapp 3000 series arrays are asymmetric active/active rather than ‘true’ active/active.
Knowledge
- Identify vCenter Server log file names and locations
- Identify ESX/ESXi log files names and locations
- Identify tools used to view vSphere log files
Skills and Abilities
- Generate vCenter Server and ESX/ESXi log bundles
- Use vicfg‐syslog to configure centralized logging on ESX/ESXi Hosts
- Test centralized logging configuration
- Configure the vMA appliance as a log host
- Use vilogger to enable/disable log collection on the vMA appliance
- Use vilogger to configure log rotation and retention
- Analyze log entries to obtain configuration information
- Analyze log entries to identify and resolve issues
Tools & learning resources
I’m covering the troubleshooting objectives last while preparing for the VCAP-DCA – it seems like the logical thing to do. Learn all the material then play with it, break it, fix it, recreate it etc. Practice makes perfect! I’ve been using the Trainsignal’s Troubleshooting for vSphere course but the official VMware Troubleshooting course has been getting good feedback.
vCenter log files
Located in;
- %ALLUSERSPROFILE%\Application Data\VMware\VMware VirtualCenter\Logs (W2k3)
- C:\ProgramData\VMware\VMware VirtualCenter\Logs (W2k8)
Available logs;
- sms.log Storage Management Service
- vpxd-xxxx.log vCenter logs
- vpxd-xxxx.log.gz are archived logs. You have to unzip them to see contents.
You can change the logging level (which defaults to ‘normal’) by going to vCenter Server Settings -> Logging Options. This VMwareKB describes how to enable trivia logging in vCenter (even if vCenter isn’t running) although this may have a performance impact and should only be used temporarily while diagnosing issues.
There are numerous ways to do this; Read more…
Security is a large topic and one you could spend a lifetime mastering. The blueprint isn’t too helpful in clarifying what level of detail you’re expected to know for this as the ESX/ESXi configuration guides cover issues not in the ‘skills and abilities’ section. More in depth still is the vSphere Hardening Guide. I guess the main thing is to focus on practical issues as the VCAP-DCA is a practical exam – knowing that the VMkernel uses memory hardening is no use in an exam if it can’t be configured or tweaked! Some of this section seems to have been added for the sake of it – how often will an admin need to modify the SSL timeouts? I could only fine one KB article about it!
Knowledge
- Identify configuration files related to network security
- Identify virtual switch security characteristics
Skills and Abilities
- Add/Edit Remove users/groups on an ESX Host
- Customize SSH settings for increased security
- Enable/Disable certificate checking
- Generate ESX Host certificates
- Enable ESXi lockdown mode
- Replace default certificate with CA‐signed certificate
- Configure SSL timeouts
- Secure ESX Web Proxy
- Enable strong passwords and configure password policies
- Identify methods for hardening virtual machines
- Analyze logs for security‐related messages
Virtual switch security characteristics
vSwitch security (layer2) settings (can be overridden at portgroup level);
- Promiscuous mode – needed for packet sniffing, vShield Zones (and virtual ESX hosts). Disabled by default.
- MAC address changes –affects inbound traffic. May need to be enabled if you’re using MS load balancing in Unicast mode, or the iSCSI software initiator with certain storage arrays. Enabled by default.
- Forged transmits – affects outbound traffic. Enabled by default.
Other network security measures (IPSec, VLANs, PVLANs etc) are dealt with in section 2, Networking.
Host security
Customise SSH settings (ESX only)
- Edit /etc/ssh/sshd.conf and set ‘PermitRootLogin’ to YES (default is NO). See VMwareKB for a list of other settings you can adjust (including the available ciphers).
- You can use PKI to authenticate using SSH without being prompted for a password. This is a standard Linux procedure – for step by step instructions see VMwareKB1002866.
- By default only SSH server is enabled. Configuration -> Security Profile to enable SSHClient, or use ‘esxcfg-firewall –e SSHClient’.
Read more…
Categories: VCAP, Virtualisation, VMware Tags: certificates, certification, esx, esxi, lockdown, passwords, security, ssl, vswitch
The blueprint for this section seems to refer mainly to ESX but I’ve described both ESX and ESXi on the assumption the lab environment used for the exams will move to v4.1 sooner rather than later.
NOTE: Weasel is VMware’s scripted installer. It’s similar to Kickstart as used with Linux, but not identical.
A summary for a scripted install;
- Decide on the bootloader source
- Configure a media repository to hold your source files and scripts
- Create an install script (either from scratch or from a previously built host)
- Perform the scripted install
Use cases for scripted installations
Reasons to use a scripted install;
- Reduce deployment time
- Ensure consistency, reduce human error
- Remove need for local media (when using PXE boot. Very useful for blade and remote environments)
- Delegate installations to junior staff who don’t know how to configure ESX
Along with knowing why you might use a scripted install in the first place you should consider the various types of scripted install and when to use each one. Factors to consider;
- Maintainability. Over time you’ll want to update your install for new releases of ESX, patches, post install steps etc. While a custom CD has the least dependencies it’s harder to maintain compared to a network media repository.
- Dependencies. I created an NFS based install only to find that most of the time the host’s physical networking hasn’t been completed when we want to build the OS, rendering this method useless. I had to convert it to a custom CD instead which was mounted via ILO (it was a blade environment). Another example is USB flash – it’s easier than CD to amend/update but may not be as useful for remote installs.
Read more…
While the blueprint only refers to installing ESX (not ESXi) I’ve covered both in anticipation of the VCAP-DCA labs going to 4.1.
When to use a customised installation
There are plenty of reasons to use some advanced installations;
- Your hardware isn’t supported in the ‘out of the box’ setup so you need custom drivers
- You want to streamline the deployment process by building a custom install CD, including some post configuration steps, or utilising PXE boot etc
- You want to gain maximum performance from every host, which means performance and configuration tweaks (vmKernel parameters, service console memory settings etc)
Installing ESX/ESXi
- Interactive installations can be done via the GUI or text mode.
- The installer can be located on CD/DVD, USB flash or via a PXE boot. While PXE is typically used for scripted builds it can be used as a source of installation files for an interactive build.
- Scripted methods (PXE boot using HTTP, FTP, NFS are covered in section 9.2.
NOTE: Scripted installs of ESXi were only added to v4.1 – prior to that only ESX classic could be scripted.
- To install a virtual ESX host on ESX (for lab testing) you need some specific configuration tweaks – see the article at vCritical for full details.
- For 64 bit guests you must have a 64 bit chip with Intel-VT support enabled or an AMD chip of revision E or later. Wikipaedia has details and you can check using VMware’s CPU Identification Utility. You cannot run nested 64 bit VMs.
- Boot from SAN is now supported for ESXi (4.1 onwards). This includes iSCSI and FCoE for a limited set of supported adapters.
- Both ESX and ESXi v4.0 will erase all local partitions by default, including existing ESX installs and VMFS partitions (if you’re upgrading an older ESX version for example).
Read more…
Loading ...
Recent Comments