Knowledge

• Identify vCenter Server log file names and locations
• Identify ESX/ESXi log files names and locations
• Identify tools used to view vSphere log files

Skills and Abilities

• Generate vCenter Server and ESX/ESXi log bundles
• Use vicfg?syslog to configure centralized logging on ESX/ESXi Hosts
• Test centralized logging configuration
• Configure the vMA appliance as a log host
• Use vilogger to enable/disable log collection on the vMA appliance
• Use vilogger to configure log rotation and retention
• Analyze log entries to obtain configuration information
• Analyze log entries to identify and resolve issues

Tools & learning resources

• Product Documentation
  • vSphere Management Assistant Guide
  • vSphere Command?Line Interface Installation and Scripting Guide
  • vSphere Datacenter Administration Guide
• vSphere Client
• vicfg-syslog, vilogger
• Eric Sloof’s Advanced Troubleshooting presentation at the Dutch VMUG
• VMware whitepaper on Troubleshooting Performance issues

I’m covering the troubleshooting objectives last while preparing for the VCAP-DCA – it seems like the logical thing to do. Learn all the material then play with it, break it, fix it, recreate it etc. Practice makes perfect! I’ve been using the Trainsignal’s Troubleshooting for vSphere course but the official VMware Troubleshooting course has been getting good feedback.

vCenter log files

Located in;

• %ALLUSERSPROFILE%\Application Data\VMware\VMware VirtualCenter\Logs (W2k3)
• C:\ProgramData\VMware\VMware VirtualCenter\Logs (W2k8)

Available logs;

• sms.log                                   Storage Management Service
• vpxd-xxxx.log                        vCenter logs
  • vpxd-xxxx.log.gz are archived logs. You have to unzip them to see contents.

You can change the logging level (which defaults to ‘normal’) by going to vCenter Server Settings -> Logging Options. This VMwareKB describes how to enable trivia logging in vCenter (even if vCenter isn’t running) although this may have a performance impact and should only be used temporarily while diagnosing issues.

There are numerous ways to do this;

• On the vCenter server console (via RDP, ILO etc) go to Start -> Program Files -> VMware -> Generate vCenter Server log bundle (or log bundle enhanced, which contains ???)

  

• From the VI client go to File -> Export -> System logs. If you only want the vCenter logs (rather than logs for ESX/i hosts) leave all the tickboxes unchecked except the ‘Include information from vCenter…’ option

• If you’re feeling flash you can do it via PowerCLI (NOTE: This can take quite a while to run!)
  • Connect-viserver <vCenter> | get-log -Bundle -DestinationPath d:\ -Server <vCenter>

You might also want to check the VI client logfiles. They can be found at;

• C:\Documents and Settings\<username>\Local Settings\Application Data\VMware\vpx (Windows XP)
• C:\Users\<username>\AppData\Local\VMware\vpx (Vista and Windows 7)

ESX/ESXi logfiles

The logs on an ESX/i host are scattered around but most of the commonly used ones are;

• /var/log
• /var/log/vmware
• /var/log/vmware/vpxa
• /var/log/vmware/hostd

VMwareKB1021800 details the logfiles for an ESX host.

VMwareKB1021801details the logfiles for an ESXi host.

Like vCenter you can generate a log bundle for troubleshooting purposes;

• Connect the VI client to vCenter and follow the same instructions above (but select the host  or hosts you want to generate a bundle for)
• Connect the VI client directly to a host and use File -> Export -> System Logs
• Using vm-support on the ESX/i console.
• Use PowerCLI  (note the syntax is slightly different)
  • Get-Log -VIHost <hostname> -Bundle -DestinationPath d:\

Tools used to view logs

There are various ways to view logfiles both in the GUI and via command line;

• Connect the VI client to vCenter and go to Administration -> System Logs to see vCenter (not host) logs. You can search these logs using the search box in the top right.
  NOTE: By default this only shows the logs since the last restart of the vCenter service.
• Connect the VI client directly to a host and go to Administration -> System Logs.

  

• Connect to a host using SSH and use standard Linux commands – cat, more, tail, and grep
  • A useful command is ‘tail -f’ which watches a file – you see updates in real time
• Point a Web browser to a host
  • http://<hostname>/host
• Use the DCUI on an ESXi host (if not in lockdown mode). Then choose the log you want to see;

1.1.1       Centralised logging

VMware’s migration to ESXi is inevitably why this objective exists, as by default it only logs into memory so all logs are lost when an ESXi host reboots. This means everyone is going to want to centralise their logging. Trainsignal’s Troubleshooting for vSphere course has a great video for this.

There are two options;

1. Use vilogger (preferred solution)
2. Configure a syslog server (if you don’t have a vMA server or have an existing syslog server)
   1. Configure the central syslog server
   2. Configure each host to send logs to the central server

Configuring vMA using vilogger

This is by far the quickest solution – for details of using vilogger refer to section 8.3.

NOTE: Occasionally when vilogger is configured the host authentication gets logged in vCenter, resulting in hundreds of logged events every minute. If you have even a small number of hosts (I tried it with just five) it can quickly fill your vCenter database with event logs. In a lab environment using SQL Server Express (with its 4GB limit) this can quickly cause your vCenter server to go offline. The solution is to disable vilogger and cleardown the database tables (see VMwareKB1025914) and it’s also been discussed in this VMware community thread.

Configuring vMA as a central syslog server

Setting up vMA as a syslog server involves the following steps;

1. Open the vMA firewall to allow incoming traffic (and make this persistent)
2. Amend the built-in syslog service (not vilogger) to receive logs from remote hosts
3. OPTIONAL:See Simon Long’s blogpost which covers adding extra hard disks to cope with additional logging capacity.

Configuring the hosts to send logs to a syslog server

Configuring ESXi

• Using vicfg-syslog for an ESXi host (NOTE: vicfg-syslog only works for ESXi)
  • vicfg-syslog -s <hostname of syslog server>
  • Can be configured to log to any syslog server
  • Go to host Configuration -> Software -> Advanced Settings -> Syslog and specify the hostname of your syslog server

NOTE: Both vicfg-syslog and editing the Advanced Settings above modify /etc/syslog.conf under the hood although if you want to do it manually there are extra steps required – VMwareKB1016621 goes into full detail.

Configuring ESX (VMwareKB1005030)

• Editing /etc/vmware/syslog.conf for an ESX host
  • Add ‘*.*     @<IP_address_of_syslog-server>’ to the bottom of the file
• Open the ESX firewall to allow syslog traffic
  • esxcfg-firewall -o 514, udp, out, syslog
• Restart the syslog service
  • service syslogd restart

VMworld 2009 session VM3325 vSphere and ESX/I logfiles 101 & 102 also covers configuring both ESX and ESXi to log to a syslog server (around the 30 minute mark).

Analyse log entries

This is a tough objective as the format and content of every logfile is different – the best study method is to simply look at every file you can find that might be useful. A good start is watching VMworld 2009 session VM3325 vSphere and ESX/I logfiles 101 & 102 (requires free registration. Audio is crap!). Two main objectives;

• Resolve issues
  • Search for warnings, errors, etc
• Obtain configuration information
  • Need to know what you’re looking for and which logfile to search
  • Use ‘grep <string> *’ to search all files in the current directory for a given string (such as ‘warning’) if you’re not sure which logfile to search (could take a long time in large directories).

NOTE: Many logfiles are time stamped using UTC – if you’re host isn’t configured to use UTC this may make correlating events and logs difficult.