Knowledge

• Recall vicfg-* commands related to listing storage configuration
• Recall vSphere 4 storage maximums
• Identify logs used to troubleshoot storage issues
• Describe the VMFS file system

Skills and Abilities

• Use vicfg-* and esxcli to troubleshoot multipathing and PSA?related issues
• Use vicfg-module to troubleshoot VMkernel storage module configurations
• Use vicfg-* and esxcli to troubleshoot iSCSI related issues
• Troubleshoot NFS mounting and permission issues
• Use esxtop/resxtop and vscsiStats to identify storage performance issues
• Configure and troubleshoot VMFS datastores using vmkfstools
• Troubleshoot snapshot and resignaturing issues

Tools

• Product Documentation
  • ESX Configuration Guide
  • ESXi Configuration Guide
  • vSphere Command?Line Interface Installation and Scripting Guide
• vSphere Client
• vicfg-* , esxcli, resxtop/esxtop,vscsiStats, vmkfstools

There’s obviously a large overlap between diagnosing performance issues and tuning storage performance, so check section 3.1 in tandem with this objective.

Recall vicfg-* commands related to listing storage configuration

• vicfg-scsidevs
• vmkiscsi-tool
• vicfg-mpath
• vicfg-iscsi
• esxcli corestorage | nmp | swiscsi
• vicfg-nas
• showmount -e
• esxtop/resxtop
  • look for CONS/s – this indicates SCSI reservation conflicts and might indicate too many VMs in a LUN. This field isn’t displayed by default (press ‘f’ then ‘f’ again to add it)
• vscsiStats
• vmkfstools
• vicfg-module

Storage Maximums

Refer to section 1.2, Storage Capacity or read VMware’s vSphere Maximum’s white paper.

Identify logs used to troubleshoot storage issues

• /var/log/vmKernel (ESX only)
• /var/log/messages
• /var/log/dmesg
• /var/log/vmkscsid.log
• /var/log/vmware/vpxa/vpxa.log
• /var/log/hostd/hostd.log
• vCenter logs
• Use tail -f to watch /var/log/vmkernel to monitor a svMotion

Describe the VMFS file system

I suspect this objective is carried over from the VI 3.5 Enterprise exam, and hence a good place to check is these Enterprise 3.5 study notes. There’s also a useful recent blogpost by Deinos Cloud.

Be aware of VMFS alignment – Vaughn Stewart’s latest blogpost

Use vicfg-* and esxcli to troubleshoot multipathing and PSA?related issues

Refer to section 1.3.

Use vicfg-module to troubleshoot VMkernel storage module configurations

Refer to section 9.1, advanced ESX builds.

Use vicfg-* and esxcli to troubleshoot iSCSI related issues

There are various ways to configure iSCSI at the command line;

• vicfg-iscsi – the most powerful CLI tool as it can configure pretty much anything – targets, host adapters, authentication, etc
• esxcli swiscsi – typically used to configure multipathing for iSCSI. See section 1.1 for details.
• esxcfg-swiscsi – can enable, http://premier-pharmacy.com/product-category/asthma/ disable, and query status of iSCSI but not much more.
• esxcfg-hwiscsi – can enable, disable, and query status of iSCSI but not much more.

Things to check;

• Are any LUNs masked? (see section 1.1)
• Which HBAs are available?

  vicfg-iscsi –H -l

• Are the expected number of paths available?

  vicfg-mpath -l

• Are the targets correctly configured?

  vicfg-iscsi –T –l vmhba33

• Are the LUNs correctly configured?

  vicfg-iscsi –L –l vmhba33

• Is authentication correctly configured?

  vicfg-iscsi –A –l vmhba33

Troubleshoot NFS mounting and permission issues

There’s a good video on this in the Trainsignal Troubleshooting Course, but in reality there’s not much to check inside VMware – permissions are handled on the storage array or NAS server.

• Use vmkping to ensure the vmKernel has connectivity to the NAS server
• ‘esxcfg-nas –l’ to show configured NFS datastores
• For ESX hosts you may be able to use the SC to diagnose permissions issues;
  • Ensure the nfsClient service is enabled on the firewall (esxcfg-firewall –e nfsClient)
  • Ensure the service console can see the storage target using ping (the target may be on an isolated network as per best practices)
  • Use ‘showmount –e <IP of storage array>’ (showmount –e 192.168.215.33)

Use esxtop/resxtop and vscsiStats to identify storage performance issues

Refer to section 3.5.

Configure and troubleshoot VMFS datastores using vmkfstools

Refer to section 1.2.

Troubleshooting snapshots and resignaturing

Use vmware-cmd;

vmware-cmd /vmfs/volumes/LocalRAID5/testVM/testVM.vmx hassnapshot

• Ruben Garcia’s excellent post on troubleshooting snapshots
• Monitoring snapshot deletion (VMwareKB1007566)
• Consolidating snapshots – a long but very good article – (VMwareKB1007849)
• Best practices for snapshots in a VMware environment (VMwareKB1025279)

For resignaturing refer to section 1.1.

Further Reading

• TA1394 – Advanced Storage Log Analysis (VMworld 2009 session)
• Eric Sloof’s Advanced Troubleshooting presentation at the Dutch VMUG
• VMware whitepaper on Troubleshooting Performance issues
• Trainsignal’s Troubleshooting for vSphere course
• A combined multivendor post about iSCSI
• Troubleshooting SCSI reservations – VMwareKB1002293,  VMwareKB1005009
• Troubleshooting SCSI reservations
• Chad Sakacc’s blogpost about storage performance, and another about VMFS/NFS limits
• VMware article on troubleshooting storage performance
• A good presentation on troubleshooting storage by a VMware engineer