This is one objective where you definitely have to get hands on – there’s no way you’ll learn esxtop otherwise. Ideally you’ll have a real infrastructure to play with as you want hosts with memory contention, ballooning, swapping, NUMA optimisations etc so you can play with and understand the features.

Knowledge

• Identify hot keys and fields used with resxtop/esxtop
• Identify fields used with vscsiStats

Skills and Abilities

• Configure esxtop/resxtop custom profiles
• Determine use cases for and apply esxtop/resxtop Interactive, Batch and Replay modes
• Use vscsiStats to gather storage performance data
• Use esxtop/resxtop to collect performance data
• Given esxtop/resxtop output, identify relative performance data for capacity planning purposes

Tools & learning resources

• Product Documentation
  • vSphere Resource Management Guide
  • vSphere Command?Line Interface Installation and Scripting Guide
  • vSphere Client
• vSphere CLI
  • esxtop/resxtop
  • vscsiStats
• VMworld 2008 session TA1440 – ESXTOP for Advanced users
• VMworld 2009 session TA3838 – ESXTOP for Advanced users
• VMworld 2010 session TA6720 – ESXTOP for Advanced users (subscription required)

Using resxtop

Two ways of invoking;

• resxtop –server <esxi host>
• resxtop –server <vCenter server> –vihost <esxi host>

Determine use cases for and apply esxtop Interactive, Batch and Replay modes

First things first – start by watching some VMworld presentations from 2008 and 2010 (subscription required). Then read some common counters to understand from the obligatory Duncan Epping blogpost about esxtop and the full list in Interpreting esxtop statistics whitepaper.

Interactive (default. Used to check current performance)

• esxtop

Batch mode (used to gather performance statistics for further analysis)

• Run esxtop in interactive mode and configure the counter you want to monitor
• run ‘esxtop -b > <filename.csv>’ (press CTRL-C to stop) or ‘esxtop -b -n 100 -d 5’ to only monitor one hundred iterations at 5 second intervals
  NOTE: you must redirect the output using ‘>’, and you should use .CSV

Replay mode (used to analyse previously collected statistics)

• Generate the performance statistics using vm-support -S -d (duration in seconds)
• Replay them using  esxtop -R <vm-support file>

You can see screenshots of this process on Hany Michael’s blogpost.

NOTE: resxtop isn’t vifastpass aware, so you have to specify –server, –username, and –password when used from the RCLI or vMA. Also resxtop doesn’t support the replay mode – is this still true?

Using esxtop custom profiles

A custom profile is where you choose to display different statistics from the default.

1. Run esxtop in interactive mode and use ‘f’ and ‘o’ to choose and order the statistics you’re interested in.
2. Press ‘W’ and specify a filename to save to (you can override the defaults or use a new file)
3. Load you custom profile using esxtop -c <filename>

Hotkeys and fields in esxtop

Commonly used keys (note these are case sensitive);

h – show help	o – order fields	s – set refresh interval
f – choose which fields to display	W – save settings	e – expand an entity
V – view only VM stats

Using vscsiStats

Prior to vSphere v4.0u2 there was very little information available about NFS storage performance, which is how I became familiar with vscsiStats (which provides storage profiles). It’s useful for profiling the storage I/O, regardless of the storage protocol used (see section 1 for details on storage workloads). Read the official vscsiStats manual. Note;

• Only available locally on an ESX/i host, not via RCLI or vMA.
• Not included on vSphere v4.0 hosts
• Included by default on vSphere v4.1 hosts

vscsiStats Theory

• Latency – anything above 15-20ms may indicate performance issues with underlying storage array.
• Seek distance – measures sequential vs random.
  • Values to the extreme sides of the histogram imply random access, more towards the middle implies sequential
  • Sequential = good, random = bad (generalisation!)
  • IO size – the stripe size on the array may be optimised accordingly
  • Read/write ratios

Gathering the above information lets you make informed decisions about which underlying RAID level may be most appropriate (depending also on the storage array).

Using vscsiStats – process

1. Find the world ID of the VM you’re interested in profiling
2. Start monitoring the VM’s storage (and optionally a specific disk)
3. Display statistics You must do this before stopping monitoring.
4. Stop monitoring

Using vscsiStats – the syntax

1. vscsiStats -l
2. vscsiStats -s -w <world ID>
3. vscsiStats -p all -w <world ID>
4. vscsiStats -x

NOTE: Follow @virtualirfan on twitter – he wrote vscsiStats! Read his VMworld 2007 presentation.

Check Duncan Epping’s blogpost for some screenshots of vscsiStats in action and Gabe’s blogpost for a full walkthrough. Michael Poore also has an interesting post on using Microsoft Chart Controls to visualise vscsiStats data and Eric Zandboer even has cool 3d Excel charts!