NVRAM problems on Netapp 3200 series filers
UPDATE FEB 2012 – Netapp have just released a firmware update for the battery and confirmed that all 32xx series controllers shipped before Feb 2012 are susceptible to this fault. You can read more (including instructions for applying the update – it’s NOT click, click, next) via the official Netapp KB article. I’ll be applying this to my production controllers soon so I’ll let you know if I encounter any problems.
Recently (Dec 2011) I’ve been experiencing a few issues with the newer Netapp filers at my work, specifically the 3240 controllers. There is currently a known issue with NVRAM battery charging which if you’re not aware of can result in unplanned failovers of your Netapp controllers. This applies to the 3200 series (including the v3200 and SA320).
We have six of these controllers and my first warning (back at the beginning of November) was an autosupport email notification;
|Symptom:||BATLOW:HA Group Notification from <myfilername> (BATTERY LOW) WARNING|
This message indicates that the NVRAM or NVMEM battery is below the minimum voltage required to safeguard data in the event of an unexpected disruption.
If the system has been halted and powered off for some time, this message is expected.This message repeats HOURLY as long as NVRAM or NVMEM battery is below the minimum voltage, if you are using ONTAP version 7.5, 8.1, or greater with an appliance that uses an NVMEM battery, the error will repeat WEEKLY.
When the storage controller is up and running, the battery will be charged to its normal operating capacity and this message should stop. However, if this message persists, there may be a problem with the NVRAM or NVMEM battery.
This was unexpected but a faulty backup battery wasn’t an immediate priority – after all it’s only required to protect against power failures or controller crashes which are pretty rare. A few days later it became a high priority after the controller failed over unexpectedly. This failure was actually triggered by the low battery level and is expected behaviour as documented in Netapp KB2011413 though it’s not made overly clear that a controller shutdown is the default action if the battery issue persists for 24 hours. I logged a call with Netapp but they were unaware of any systemic issues and despite pointing out that this was affecting all six of our controllers they simply sent replacement NVRAM batteries and suggested we swap them all out. I posted a question on the Netapp forums but at the time no-one else seemed to be having the same issue. The new batteries were duly fitted and the problem seemed to be resolved – I’ve since rechecked our battery charges and they’re stable at around 150 hours.
An update in an email we received from Netapp on the 22nd December now states that it’s a known firmware issue with a permanent fix currently expected in Feb 2012. Netapp advise that further downtime will be required to implement the fix when it’s made available.
Don’t ignore low battery alerts!
Are your Netapp’s affected?
Even if you haven’t had any autosupport warnings you should still check your controllers to ensure the NVRAM battery is fully charged. You can do this from the command line;
There are three settings you look for;
- the ‘Bat Run Time’ entry which should show either ‘OK’ (for older 3000 series) or a number of hours (for 3100/3200 series). This should show around 140 hours if they’re fully charged.
- the ‘Charger Volt’ setting should be 8.2V. If the setting is ’0′ then your battery isn’t charging.
- the ‘Charger Current’ setting should be 2000ma. If the setting is ’0′ then your battery isn’t charging.
What’s the fix?
Unfortunately the fix requires shutting down the affected controllers. For most people using clustered controllers this won’t be a major issue but for those running NFS (which has a longer potential recovery time from handovers and failbacks) you may need to schedule downtime. If you run Oracle RAC (10g or 11g) you’ll need to amend the RAC configuration (CSS timeout) to prevent node failures.
See Netapp bug 536445 for a full description and suggested workaround.