While the blueprint only refers to installing ESX (not ESXi) I’ve covered both in anticipation of the VCAP-DCA labs going to 4.1.
When to use a customised installation
There are plenty of reasons to use some advanced installations;
- Your hardware isn’t supported in the ‘out of the box’ setup so you need custom drivers
- You want to streamline the deployment process by building a custom install CD, including some post configuration steps, or utilising PXE boot etc
- You want to gain maximum performance from every host, which means performance and configuration tweaks (vmKernel parameters, service console memory settings etc)
Installing ESX/ESXi
- Interactive installations can be done via the GUI or text mode.
- The installer can be located on CD/DVD, USB flash or via a PXE boot. While PXE is typically used for scripted builds it can be used as a source of installation files for an interactive build.
- Scripted methods (PXE boot using HTTP, FTP, NFS are covered in section 9.2.
NOTE: Scripted installs of ESXi were only added to v4.1 – prior to that only ESX classic could be scripted. - To install a virtual ESX host on ESX (for lab testing) you need some specific configuration tweaks – see the article at vCritical for full details.
- For 64 bit guests you must have a 64 bit chip with Intel-VT support enabled or an AMD chip of revision E or later. Wikipaedia has details and you can check using VMware’s CPU Identification Utility. You cannot run nested 64 bit VMs.
- Boot from SAN is now supported for ESXi (4.1 onwards). This includes iSCSI and FCoE for a limited set of supported adapters.
- Both ESX and ESXi v4.0 will erase all local partitions by default, including existing ESX installs and VMFS partitions (if you’re upgrading an older ESX version for example).
USB flash devices can be used in two ways;
- You can run ESXi (but NOT ESX classic) directly from an attached USB drive (provided your BIOS sees it as bootable). To do this simply choose the USB drive as the install point when prompted by the ESX installer.
- You can use the USB drive as the source for installation media and scripts
Service console defaults and maximums
Default is 300MB for servers with up to 8GB RAM then scales towards the maximum of 800MB for a host with more than 128GB RAM. VMware recommend you don’t change manually, although it is generally recommended practice to size your SWAP partition to 1600MB so that you don’t need to rejig partitions when increasing the memory size at a later date.
This is all documented on Duncan Epping’s Yellow Bricks site.
Configure optional partitions during install
- Only ESX classic has a user definable partition layout – ESXi uses an automatic partitioning scheme which the user can’t change. See this blogpost at Jason Boche’s site, and another blog article at Geeksilver’s site (and one more for luck).
- From the installation guide – “You cannot define the sizes of the /boot, vmkcore, and /vmfs partitions when you use the graphical or text installation modes. You can define these partition sizes when you do scripted installations”. See the ESX and vCenter Server Installation Guide (chapter 6) for full details.
- NOTE: the ‘/boot’ partition used to be only 100MB (with ESX 3.5 and prior) but has now been increased (to 1.25GB) to allow a future upgrade from ESX to ESXi
Mount point | Default size | Recommended size | Filesystem type | Physical location |
/boot | 1100MB | 1100MB | Ext3 | Physical partition |
vmkcore | 100MB | 100MB | vmkcore | Physical partition |
n/a | 1.2GB+ | 1.2GB+ | VMFS | Physical partition which fills any remaining space on the physical disk |
The following are all in the VMFS volume created during install | ||||
/ | 5GB+ (various depending on disk size) | Leave as default | Ext3 | VMDK in the VMFS volume |
esxconsole.vmdk | 1.2GB | Leave as default | Ext3? | VMDK in the VMFS volume |
swap | 600MB | 1600MB | swap | VMDK in the VMFS volume |
/var | 2GB (/var/log) | 5GB | ext3 | |
/opt | Optional | 2GB | ext3 | |
/home | Optional | 512MB | ext3 | |
/tmp | Optional | 1GB | ext3 | |
/usr | Optional | None specified | ext3 |
Configure advanced bootloader options
- Can be used to set a password on the bootloader (which is then requested if you want to change kernel parameters at boot time. See this article on securing ESX for details)
- By default the GRUB bootloader is installed in the MBR (master boot record). Some legacy hardware stores BIOS info in the MBR so in these cases you have to install GRUB on the first partition of the disk instead.
- Allows you to specify kernel parameters, which are written to the GRUB.CONF file to ensure they’re http://www.eta-i.org/tramadol.html persistent across reboots. I couldn’t find much documentation on how these are used with ESX – Eric Sloof has a post around possible kernel parameters with ESX3i.
Configure kernel parameters
Typically used with scripted installs (see section 9.2).
Install/uninstall custom drivers
There are occasions when required drivers are not included in the ESX or ESXi builds (for example the HP 375T quad port 10GB NIC which requires custom drivers for the HP BL460c G6 and G7 blades). With vSphere you can now add drivers either during installation (ESX only) or post install (both ESX and ESXi) although the procedure is different for each;
At install time
- Start the installation as usual – this can be interactive or scripted
- When prompted, reply Yes to ‘install additional drivers?’. You can either embed them in a custom ESX .ISO or provide a separate CD.
- NOTE: You can’t add custom drivers when using a PXE install (see chapter 3 of install guide)
Post-install
- Check online HCL to determine the driver required. Provides link to .ISO.
- Download drivers from VMware, check signature using md5sum <drivers.ISO>
- Use esxupdate for ESX classic hosts or vihostupdate (vMA, vCLI etc) for both ESX and ESXi hosts
- esxupdate –bundle <drivers-file.zip> update
- vihostupdate [options]—install –bundle <drivers-file.zip>
- Typically a reboot of the host is required (this is normally indicated in the release notes). Some bundles will require the host to be in maintenance mode before application – this will be indicated where necessary.
Instructions for this are summarised in this VMware post, or you can read more in the VMware Patch Management Guide.
Querying existing drivers;
- esxupdate query –vib-view (used to get the driver name)
- ethtool –I vmnicX (shows the driver version in use. This only works for ESX. For an ESXi version refer to VMware KB 1027206)
Uninstalling a driver uses a similar syntax;
- esxupdate –-bundle <driver filename> remove (for example esxupdate –-bundle tg3 remove)
NOTE: Some driver updates are provided in a metadata file rather than a bundle. This is simply an XML file which point to the actual bundle to use. When using metadata use –-metadata not –-bundle in the command.
Post install configuration
Despite the functionality offered during the install there is often further configuration required. The tools available to do this vary depending on the deployment scenario;
- PowerCLI – can’t run natively during scripted installs but can be used via a server based component which ‘listens’ for a completed build and finishes the post build tasks. Links?
- vCLI – any vCLI commands can be included in %pre and %post
- Host profiles (see chapter 5, Operations Maintenance)
vicfg-module
This is used to configure advanced VMkernel options. Typical uses;
- to set queue depth for HBAs (VMware KB1267)
- enabling Netqueue (see blueprint section 2.1 on Networking for details)
- for fixing faulty drivers (see VMware KB1029070). This will also be relevant to section 6.4, troubleshooting storage performance and connectivity.
Syntax
vicfg-module [<connection_options>]
[–get-options <module_name> |
–help |
–list |
–set-options “<option> <value>” <module_name> |
–vihost <esx_host> ]
NOTE: Setting options via esxcfg-module is NOT cumulative. If you only specify one option in the command it will clear any other previously set parameters.
Examples
vicfg-module –get-options
Shows the enabled options for a module. NOTE: This doesn’t show the available options, only the enabled ones. To get a list of possible values use the older vmkload_mod –s <module>.
vicfg-module –d vmfs2
Disable the module, preventing it from reloading after a reboot. Can be used in conjunction with –u.
vicfg-module –u vmfs2
Unload the module immediately. Would potentially be re-enabled at reboot unless the ‘-d’ option was also used.
vicfg-advcfg
This is a vCLI command to configure advanced parameters on ESX and ESXi hosts, equivalent to the host Configuration/Advanced settings you’d configure with the VI client. Typical uses;
- enable or disable CIM providers (ESXi)
- configure a host during a scripted build – set NFS options as per this blogpost from Xtravirt
Syntax
vicfg-advcfg <connection_options>
[–default <value> |
–get <path> |
–get-kernel <boot_parameter> |
–quiet |
–help |
–set <value> <option> |
–set 0|1 UserVars.CIMEnabled |
–set 0|1 UserVars.CIMOEMProvidersEnabled |
–set 0|1 UserVars.CIMCustomProvidersEnabled |
–set-kernel <value> <boot_parameter> |
–set-message <message> |
–list
–vihost <esx_target>]
Examples
vicfg-advcfg –server esx01.vExperienced.co.uk –get LVMDisallowSnapshotLun
vicfg-advcfg –server esx01.vExperienced.co.uk –set 2 LVMDisallowSnapshotLun
vicfg-advcfg –server esx01.vExperienced.co.uk –get-kernel
vicfg-advcfg –server esx01.vExperienced.co.uk –set-kernel
Links to find out more;
http://www.vm-help.com/esx/esx3i/esx_3i_rcli/vicfg-advcfg.php
http://it-john.com/home/technology/vmware/esxcfg-advcfg/
Further reading
Custom ESX CD/DVD – if you want to have the flexibility of a scripted install combined with a custom ESX CD there are various sites with instructions (here and here).
One thought on “VCAP-DCA Study notes – 9.1 Installing ESX with custom settings”