Patch-ID# 113723-07 Keywords: storedge 3510 se3510 se3000 raid storage firmware ses pld Synopsis: SE3510 327R: StorEdge 3510 array controller firmware upgrade Date: Mar/22/2004 Install Requirements: None Solaris Release: 8 9 SunOS Release: 5.8 5.9 Unbundled Product: StorEdge 3510 Array Controller Firmware Unbundled Release: 327R Xref: Topic: Relevant Architectures: sparc BugId's fixed with this patch: 4784958 4787696 4805923 4808697 4810830 4825913 4826287 4832000 4832464 4833731 4838275 4849102 4849745 4859259 4860383 4862475 4865012 4867584 4880182 4881132 4881678 4889104 4892830 4893753 4893756 4900378 4900579 4906151 4908244 4908349 4916011 4932155 4932175 4939014 4942103 4942158 4952788 4955208 5012640 Changes incorporated in this version: 4860383 4867584 4880182 4881678 4893753 4893756 4900378 4900579 4906151 4908244 4908349 4916011 4932155 4932175 4939014 4942103 4942158 4952788 4955208 5012640 Patches accumulated and obsoleted by this patch: Patches which conflict with this patch: Patches required with this patch: Obsoleted by: Files included with this patch: README.113723-07 fc2u_sun_1040a.s3r # SE3510 Enclosure SES Firmware rev 1040 patchinfo pld1r10.s3r # SE3510 Enclosure PLD Firmware rev 1000 SUN327R-3510.bin # SE3510 Array Controller Firmware rev 327R Problem Description: This patch contains all three types of firmware updates for the StorEdge 3510 array subsystem: firmware for the array RAID controllers; firmware for the SES processors in each enclosure; and firmware for the programmable logic devices (PLDs) that route Fibre Channel data within each enclosure. The SES and PLD firmware updates are applicable to the StorEdge 3510 FC Array expansion chassis (JBOD) as well as the FC RAID Array chassis. If you are upgrading firmware in a standalone FC JBOD Array, you will skip the RAID controller firmware instructions but follow instructions for the SES and PLD firmware upgrades. You do not need to install this patch if you have all of the following firmware revision levels in all of your StorEdge 3510 enclosures: Controller firmware: 327R SES firmware: 1040 PLD firmware: 1000 You do need to install this patch if you have any of the following: * controller firmware earlier than revision 327R, or * SES firmware A100 or 1000, or * PLD firmware A000 If you're not sure how to determine what firmware you currently have installed, follow the instructions in the Patch Installation Instructions section. RAID firmware 327R addresses the following issues: 4860383 out of band cli interface hangs during diag channel inquiry after 1 to 4 hours 4867584 missing LUNs on reboot (1024 LUNs, 512 per channel) 4906151 lack of sccli oob functionality follows raid controllers 4916011 Parity check never completes on random LD's in certain configurations. 4932155 P2S2 I/O failure(giving up) due to command timeout on datafailover test 4939014 Controller hang when Controller failed through sccli 4952788 telnet sessions to 3510 hang (DUPLICATE of 4860383) 4955208 Primary controller fault lead to host IO and LUN failure 5012640 Disk side fibre channel CRC link errors may cause the controller to fail SES Firmware 1040 addresses the following issues: 4908349 FRU physical location within 3510 is not available from the SCCLI. 4880182 FCP response packet sometimes fills in non-zero values in reserved fields (from 113723-06) RAID firmware 327Q addresses the following issues: 4893753 FC 3510 continuously returns STATUS_BUSY 4893756 FC 3510 not responding to INQUIRY 4900378 SE3510 failed SSTM 3.1 24-hour Load & Stress (No error injection) 4900579 3510 Cache gets "pinned" at a certain percentage 4908244 the telnet menu shutdown reset hangs when cache is pinned from bug id 4900579 SES Firmware 1030 addresses the following issues: 4881678 Can't recover the FW status from data path failure on expansion cable This README file addresses the following issues: 4932175 Firmware patch is confusing about upgrading the PLD 4942103 Patch 113723-06, README.113723-06 has embeddded control characters. 4942158 3510 - SES upgrade to 1030, README.113723-06 has incorrect SES Upgrade Step 9. (from 113723-05) RAID firmware 327P addresses the following issues: 4859259 telnet connection drops 4865012 Filtered LUN maps persist after Logical Drive is reassigned to other controller. 4881132 Disk status is incorrect when rebuilding after RAID 1 multiple drive failures 4892830 Cntrl F/W does not Retry if Hard Drive issues timeout (62F error) (from 113723-04) RAID firmware 327N addresses the following issue in version 327M: 4862475 heavy i/o load can cause unnecessary controller failover (from 113723-03) RAID firmware 327M addresses the following issues in version 327K: 4889104 Logical drive initialization phase takes too long 4784958 key is too sensitive to select at 3.27B FW 4805923 LUNs are inaccessible after Host resets 4808697 out of band sccli interface unavailable after extended operation. 4810830 Spelling error in "view and edit Configuration parameters" menu 4832000 scsi 2 reboot test tc_mhiocstatus failure 4832464 loss of access to LUNs after node reboots 4833731 rebooting a host impacts channel availability 4849102 scsi2 tc_mhiocstatus reboot failure with LUN filtering enabled 4849745 Deassert a failed secondary controller causes data corruption in point to point PLD firmware version 1000 addresses the following issues: 4833731 rebooting a host impacts channel availability 4838275 Direct connect arrays failing scsi 3 reboot tests (from 113723-02) RAID firmware 327K addresses the following issue in version 327I: 4825913 Point to point only mode will check in as loop if more than 1 id is assigned 4805923 LUNs are inaccessible after Host resets 4787696 controller failed persistent reservation tests (from 113723-01) 4826287 copy and replace disk activity during LD rebuild causes data corruption Patch Installation Instructions: -------------------------------- This patch contains firmware which is installed on the StorEdge 3510 array, rather than using installpatch or patchadd. The prerequisites for installing the new firmware are: * Version 1.5 of the "sccli" CLI utility included in the SUNWsccli package, which may be installed from the Sun StorEdge Professional Storage Manager CD or downloaded from the Sun Download Center. The sccli command is installed, by default, in /opt/SUNWsscs/sbin on Solaris systems, and a symbolic link is provided in /usr/sbin so that the user may simply type "sccli" to invoke it. If you do not have this package installed, or if you have an older version, download and install the latest version from the Sun Download Center: 1. Go to http://www.sun.com/software/download and Click on the Systems Administration link 2. Click on the link for "StorEdge 3000 Family Storage Products--Related Software 3. Log in or register as necessary 4. Download the file 1.5.0_sw_XXX.zip where "XXX" describes your platform. 5. Unzip the file and install the package. For example, on Solaris: pkgrm SUNWsccli # remove old version pkgadd -d solaris/`uname -p` SUNWsccli 6. Verify that the correct version is installed by typing /opt/SUNWsscs/sbin/sccli version DO NOT ATTEMPT TO USE AN EARLIER VERSION OF SCCLI. * For RAID controller firmware upgrades, an Ethernet connection (or two Ethernet connections for dual-controller configurations) is required for out-of-band firmware upgrades, if the array to be upgraded is not connected to the host on which the upgrades are to be performed. In order to use the Ethernet connection(s), the controller must have an IP address assigned. * A Fibre Channel connection to the array is required when no network connection to the array is available. Unpack the patch into any convenient directory (for example, /var/tmp), and refer to the Special Install Instructions below. Special Install Instructions: ----------------------------- *** Warning *** Warning *** Warning *** Warning *** Warning *** DO NOT INSTALL THIS PATCH with a version of sccli earlier than version 1.5. If an earlier version is installed, refer to the upgrade instructions above. *** Warning *** Warning *** Warning *** Warning *** Warning *** In dual-controller RAID configurations in a fabric topology, upgrading the SES firmware may alter the state of the inter-controller-links that enable controller failover to work. This may interrupt I/O activity until the state of the links is restored manually. If upgrading the SES firmware in a dual-controller RAID in a fabric topology where the inter-controller-links have been previously disconnected, follow the precautions in the next paragraph and assume that the array will be unavailable for several minutes following the SES firmware upgrade. Be sure to follow the instructions in steps 7 and 12 to preserve your configuration and restore it after the SES update is completed. *** Warning *** Warning *** Warning *** Warning *** Warning *** The controller firmware procedures below will prevent the StorEdge 3510 array from servicing normal I/O requests for several minutes if the controller is currently running 327I or 327J firmware, even in redundant controller configurations that support live firmware upgrades. The PLD firmware upgrade will also interrupt I/O, and will require a power-cycle of the enclosure. To avoid any possibility of application failures or filesystem corruption, it is necessary to ensure that no host I/O is permitted while the upgrade is taking place. For this reason, all StorEdge 3510 logical disks should be unmounted, from any Solaris hosts connected to the array, for the duration of the install procedure. For example, filesystems mounted from the StorEdge 3510 should be unmounted, and databases accessing raw devices on the array should be shut down. *** Warning *** Warning *** Warning *** Warning *** Warning *** If the array controller is power-cycled at the wrong time during programming of the PLD firmware, the array controller may be rendered inoperable and require replacement. This procedure upgrades three different firmware files for three different hardware components: the RAID controller, which processes I/O requests; the SCSI Enclosure Services (SES) processor, which handles enclosure management functions; and a Programmable Logic Device (PLD) which is responsible for routing of Fibre Channel traffic within the subsystem. Preparing to Upgrade Firmware: ------------------------------ 1. Change your working directory to the directory in which the patch was unpacked and confirm that the SUN327R-3510.bin file is present using the "ls" command. If upgrading a StorEdge 3510 RAID controller chassis, continue with step 2. If upgrading a StorEdge 3510 standalone expansion chassis (JBOD) which is directly attached to the host, you must identify two different SES device filenames for the chassis, and repeat the instructions in steps 2 through 18 twice, once for each SES device. This is because there are two internal loops in each enclosure, and each loop contains its own SES and PLD devices, located on an I/O Module FRU which is used to provide host connections for that loop. Because the SES and PLD devices are redundant, the SES firmware and the sccli utility normally hide the fact that there are two separate devices from the user, and sccli by default will only display one of them at a time. However, it is important to ensure that both sets of SES and PLD devices are upgraded at the same time, so it is currently necessary to identify the individual SES devices and perform the upgrade procedure separately on each device. To identify the SES devices, run the following command: for d in /dev/es/*; do sccli --list $d; done | grep 3510F >jbods.txt This will run the command "sccli --list /dev/es/" for each device filename found in /dev/es, and save the lines that match the FC JBOD Array to the file "jbods.txt". This command may take a few moments to complete and may display some harmless error messages referring to non-existent devices that may safely be ignored. When it completes, jbods.txt should contain two lines for each directly-attached JBOD chassis. For example, if jbods.txt contains: /dev/es/ses3 SUN StorEdge 3510F D SN#000187 /dev/es/ses9 SUN StorEdge 3510F D SN#000187 it indicates that /dev/es/ses3 and /dev/es/ses9 are two separate SES devices within the JBOD expansion chassis with serial number 000187. For each device filename found, verify that you see two device filenames for each chassis serial number, indicating that you've identified a device filename for each of the two SES devices in that chassis. If a chassis appears only once in the list, verify that that chassis contains only one I/O Module FRU. If a chassis with two I/O Modules is represented by only one device filename, refer to the special instructions under the heading "Non-Redundant FC JBOD Array Upgrades." The procedure outlined in steps 2 through 18 should be repeated for each device in the left column. 2. Invoke sccli, using the appropriate syntax for the type of connection used to manage the array: If the array to be upgraded is a standalone expansion chassis (JBOD), invoke sccli with the name of an SES device identified in step 1: sccli /dev/es/ses If the array to be upgraded is a RAID subsystem attached to a Fibre Channel HBA installed in the local host host, simply type the sccli command with no arguments: sccli If more than one array is connected to local Fibre Channel HBAs, sccli will display a list from which the appropriate device may be selected. If the array has an IP address configured, it may be managed remotely using a network (or "out-of-band") connection. To specify this mode, type: sccli [-w ] where is the password assigned to the controller, and is the hostname or IP address of the array. Use single quote characters to quote the password if it contains any characters that are interpreted specially by the shell. If no controller password is set, the -w option and password argument should be omitted. If the hostname for a locally attached array is unknown, the "--oob" option may be specified instead of a hostname, and sccli will try to determine the IP address of a locally-attached array automatically. Verify that you're running sccli version 1.5 with the "version" subcommand. 3. Verify the Product and Revision of the array. Type the command show inquiry and examine the output. If the "Product" is "StorEdge 3510F D" and "Device Type: Enclosure" appears, the device is a JBOD chassis; proceed to the SES firmware upgrade at step 7. If the Product is not "StorEdge 3510", this patch does not apply. Select a different device or discontinue installation of this patch. If the firmware version reported is 327R or higher, the RAID controller firmware does not need to be upgraded; proceed to step 7. If it is reported as "3510 327I vX.XX", or 327 which is less than 327R, continue with step 4. 4. If array contains only one controller, or the firmware revision is earlier than 327K, or the "show redundancy" command does not report that the array redundancy mode is Active-Active and its redundancy status is "Enabled", the controller firmware upgrade will require that the controller be reset, which in turn will cause the controller to stop service I/O requests for several minutes. If one of the previous conditions is true, skip to step 5 and follow the instructions for on upgrading a single-controller configuration. If you have a dual controller configuration running firmware 327K or higher, and the "show redundancy" command reports that the Redundancy mode is "Active-Active" and that the status is "Enabled", perform a live firmware upgrade by typing the following command at the CLI prompt: download controller-firmware SUN327R-3510.bin The sccli command will display messages indicating that it is downloading the firmware, programming the controller's flash memory, and "engaging" the new firmware. Wait until the "sccli>" prompt appears again, or 3-4 minutes, before proceeding. If you see an error message containing the text: "please repeat download command" or "controller firmware download ok, but engage operation is not allowed", the controller was unable to perform a live firmware upgrade. Wait until it is OK to stop I/O to the array, and then follow the instructions in step 5 to do a non-live firmware upgrade. If no error messages were displayed, proceed with step 6. 5. Single controller configurations, dual controller configurations which are in degraded status, or controllers running firmware earlier than 327K must be reset after downloading new firmware. First, stop all I/O activity and unmount any filesystems or volumes mounted from the array. Then, at the "sccli>" prompt, type: download controller-firmware -r SUN327R-3510.bin You should see output indicating that the firmware is being downloaded and programmed into flash memory, and then a message indicating that the controller is being reset. Wait until the "sccli>" prompt appears before proceeding. 6. Verify that the firmware upgrade succeeded by examining the firmware revision again. At the "sccli>" prompt, type show inquiry and verify that the firmware revision is now reported as "327R". SES Firmware Upgrade: --------------------- 7. Verify the SES firmware version using the sccli "show ses" subcommand: sccli> sho ses Ch Id Chassis Vendor/Product ID Rev PLD WWNN ... ------------------------------------------------------------------ 2 12 000187 SUN StorEdge 3510F A 1030 A000 204000C0FF000187 ... Topology: ... 3 12 000187 SUN StorEdge 3510F A 1030 A000 204000C0FF000187 ... Topology: ... In the example above, the information that appears where "..." is shown above may not be accurate until after the firmware update is completed, and it's not needed for the upgrade, so it won't be shown here. Note that only one line of information may be displayed for a standalone expansion (JBOD) chassis, and more than two lines may be displayed if a RAID enclosure has expansion chassis connected to it. The Product ID field will be "StorEdge 3510F A" for SES devices internal to the RAID array chassis, and "StorEdge 3510F D" for SES devices internal to the expansion chassis. Examine the SES firmware revision codes displayed in the "Rev" column. If all of the lines contain a revision code of 1040 in the "Rev" field, then no SES firmware update is necessary and you should proceed to step 13. In this example, the "Rev" field is 1030 for both devices and therefore an upgrade is needed. If upgrading a standalone expansion chassis (JBOD) enclosure, continue with step 9. 8. Record the state of the inter-controller-links. At the "sccli>" command prompt, type: show configuration old-configuration.txt to save the configuration of the array to a file called "old-configuration.txt". Note that this command may take several minutes to complete. This file may be referred to later in case it is necessary to restore the state of the inter-controller-links. 9. Since the SSCS agent may interfere with download of SES code, it is necessary to stop any SSCS agent that may be running on a host connected to the array controller. To stop the SSCS agent, issue the shell command "/etc/init.d/ssagent stop" as root on any system where the SUNWscsa package is installed. If the SSCS agent is running on a Microsoft Windows system connected to the array, use the "Services" application (Start->Run->"services.msc") to stop the SSCS Agent service. StorADE, or any other application which monitors the status of the enclosure, may also interfere with controller or SES firmware downloads. Stop the StorADE agent software, or any other software that monitors the array, following the procedures described in the documentation for that application. 10. Invoke sccli again. At the "sccli>" prompt, type the following command: download ses-firmware fc2u_sun_1040a.s3r sccli performs a "live upgrade" of the SES firmware and it is not necessary to restart the RAID controller afterwards; however, until the next controller reset, the firmware's telnet/RS232 menu interface may continue to display the old SES firmware revision. 11. Run the sccli subcommand "show ses" again to verify that all SES devices have been upgraded to version 1040, as indicated by the "Rev" column. In the example below, the Rev column has changed to "1040" for both SES devices: sccli> sho ses Ch Id Chassis Vendor/Product ID Rev PLD WWNN ... ------------------------------------------------------------------ 2 12 000187 SUN StorEdge 3510F A 1040 A000 204000C0FF000187 ... Topology: ... 3 12 000187 SUN StorEdge 3510F A 1040 A000 204000C0FF000187 ... Topology: ... Note: While it is not necessary to reset the RAID controller after an SES firmware upgrade, the RAID controller's telnet/RS232 menu interface may continue to display the old SES revision number until after the next time it is reset. 12. If you are upgrading a standalone expansion chassis, skip to step 13. If upgrading RAID enclosure, type: show inter-controller-link at the "sccli>" prompt, and verify that all links are "connected" (the default state). If you have difficulty communicating with the controller using an inband connection, try using an out-of-band (network) connection instead. If some links are not connected, or if using a point-to-point configuration where the inter-controller-links should be disconnected, you will need to set the connection state again. Refer to the "inter-controller-link" section of the configuration file you saved in step 8 for the settings to be restored. If a link that was previously disconnected has become connected, it can be disconnected again with the commands: set inter-controller-link upper channel disconnected set inter-controller-link lower channel disconnected If a link that was previously connected has become disconnected, it can be connected again with the commands: set inter-controller-link upper channel connected set inter-controller-link lower channel connected where is the channel that should be connected. PLD Firmware Upgrade: --------------------- 13. Warning: Performing this PLD firmware upgrade incorrectly, by using a version of sccli prior to version 1.5 or power-cycling the array at the wrong time, could result in the array enclosure being rendered unusable. Upgrading the PLD firmware causes the host's Fibre Channel HBAs to be disconnected from the array temporarily, which may prevent sccli from completing the procedure properly if sccli is not used out-of-band. sccli must be invoked with the hostname or IP address assigned to the array, so that a network connection is used to communicate with the array controller (see Step 2). The PLD firmware upgrade cannot be completed without power-cycling the array at the end of the procedure. Do not proceed if it is not possible to power cycle the array at this time. 14. Verify the PLD firmware revision by issuing the sccli "show ses" subcommand: sccli> sho ses Ch Id Chassis Vendor/Product ID Rev PLD WWNN ... ------------------------------------------------------------------ 2 12 000187 SUN StorEdge 3510F A 1040 A000 204000C0FF000187 ... Topology: ... 3 12 000187 SUN StorEdge 3510F A 1040 A000 204000C0FF000187 ... Topology: ... Examine the PLD firmware revision codes displayed in the "PLD" column. If all of the lines contain a revision code of 1000 in the PLD column, then no PLD firmware update is necessary and you should proceed to step 20. 15. Make sure that all host I/O activity has been stopped. The array will be unable to process host I/O activity for up to 10 minutes. 16. Type the following command at the "sccli>" prompt: download pld-firmware pld1r10.s3r sccli will indicate that it is shutting down the controller, sending the PLD firmware data, and programming the flash memory. All the LEDs on the array will illuminate while the flash programming is under way, and sccli will pause for 3 to 5 minutes. DO NOT POWER CYCLE THE ARRAY YET. Note the time and wait at least 5 minutes for the PLD flash programming operation to complete. 17. After waiting at least 5 minutes after seeing the message "Please wait 3 to 5 minutes until PLD flashing completes", turn off power to the array, wait 10 seconds, and then turn power back on. Be sure to turn off, or unplug, BOTH power supplies, as removing power from one power supply does not cause the array to lose power. Allow up to 3 minutes for the controller to restart fully. 18. Restart sccli and run the "show ses" command again to verify that the PLD firmware revision displayed in the PLD column is 1040 for all SES devices shown, as in the example below: sccli> sho ses Ch Id Chassis Vendor/Product ID Rev PLD WWNN ... ------------------------------------------------------------------ 2 12 000187 SUN StorEdge 3510F A 1040 1000 204000C0FF000187 ... Topology: ... 3 12 000187 SUN StorEdge 3510F A 1040 1000 204000C0FF000187 ... Topology: ... 19. If you are upgrading a RAID controller array, continue with step 20. If you are upgrading a standalone expansion chassis, repeat steps 2 through 18 for the other SES device within the same chassis (identified in step 1) before returning the array to service. After Upgrading: ---------------- 20. If the SSCS agent was stopped in step 9, restart it with the command "/etc/init.d/ssagent start" if it is installed on Solaris, or using the "Services" utility (Start->Run->"services.msc") if it is installed on Microsoft Windows. If other storage management software such as StorADE was stopped, restart it following the procedures in the documentation for that software. Additional Information ---------------------- Non-Redundant FC JBOD Upgrades ------------------------------ Upgrading the two SES devices (and their associated PLD devices) in the 3510 FC JBOD Array requires access to two separate device filenames in /dev/es, one for each SES device. If you can only locate one device filename in /dev/es for an FC JBOD Array, use the following guidelines to determine how to proceed: a) If there is only one physical I/O Module installed in the FC JBOD Array: Only one SES device is present and no special preparation is necessary because no firmware revision mismatch can occur. b) If there is only one cable connection from the host to the FC JBOD Array: Upgrading the firmware in both I/O Modules requires moving the cable from one I/O Module to the other, which requires host system downtime. Do not attempt to upgrade the SES or PLD firmware in this chassis until you are able to do all of the following: * Follow the instructions in steps 2, 3, 7, and 14 (only) to determine if you need to upgrade the SES or PLD firmware in the chassis, but do not proceed to steps 8 or 15. If the SES and PLD firmware are at the correct revisions, no futher action is required for this FC JBOD Array. * Stop all I/O activity to the array, and unmount any volumes or filesystems mounted from it; reboot to single-user mode if necessary. * Unplug the cable from the I/O Module to which it's currently connected and plug it in to the other I/O Module. * Run "devfsadm" to create a new /dev/es device filename corresponding to the second SES device which is now connected to the host (the first one will no longer be accessible). * Follow steps 2 through 18 above to upgrade this second SES device. * Move the cable back to the I/O Module to which it was originally connected, and run "devfsadm" again. If desired, run "devfsadm -C" to remove the device file created for the second SES device above. * Follow steps 2 to 18 to upgrade the first SES device. * Reboot the host to remount volumes and filesystems and restore I/O activity to the array. c) If there are two cables from the host to the FC JBOD Array: Ensure that they are connected according to the instructions in the Sun StorEdge 3000 Family Installation, Operation, and Service Manual for the Sun StorEdge 3510 FC Array, Part No 816-7300-15. Run the "devfsadm" command after making any cabling changes and verify that the chassis is represented by two device filenames in /dev/es. If a second device filename does not appear, troubleshoot the connections before attempting to upgrade the FC JBOD Array firmware. README -- Last modified date: Tuesday, November 2, 2004