The other day I was alerted to a POE problem on an Avaya / Nortel ERS4548GT-PWR when trawling through a syslog server. What drew my attention was the "
PoE Main Power Usage On 512 Watts" log entry, which I had not seen before.
So I logged onto the switch and ran a few commands in an attempt to explore the problem.
ERS4500#show sys-info
Operation Mode: Switch
MAC Address: MA-CA-DD-RE-SS-00
PoE Module FW: Unknown
Reset Count: 14
Last Reset Type: Power Cycle
Power Status: Primary Power
Autotopology: Enabled
Pluggable Port 45: None
Pluggable Port 46: None
Pluggable Port 47: None
Pluggable Port 48: LX
Base Unit Selection: Non-base unit using rear-panel switch
sysDescr: Ethernet Routing Switch 4548GT-PWR
HW:12 FW:5.3.0.3 SW:v5.6.1.053
Mfg Date:20120121 HW Dev:
Serial #: SERIALNUMBER
Operational Software: FW:5.3.0.3 SW:v5.6.1.053
Installed software: FW:5.3.0.3 SW:v5.6.1.053
Operational license: Base software
Installed license: Base software
sysObjectID: 1.3.6.1.4.1.45.3.71.2
sysUpTime: 43 days, 01:22:00
sysNtpTime: 2013-05-28 04:07:52 GMT+00:00
sysServices: 6
sysContact:
sysName:
sysLocation:
Stack sysAssetId:
Unit sysAssetId:
ERS4500#show poe-main-status
% Internal error
ERS4500#show poe-power-measurement
Port Volt(V) Current(mA) Power(Watt)
---- ------- ----------- ---------------
% Internal error
ERS4500#show poe-port-status
Admin Current Limit
Port Status Status Classification (Watts) Priority
---- ------- ----------------- -------------- ------- --------
1 Enable Detecting 0 16 Low
2 Enable Detecting 0 16 Low
3 Enable Detecting 0 16 Low
4 Enable Detecting 0 16 Low
5 Enable Detecting 0 16 Low
6 Enable Detecting 0 16 Low
7 Enable Detecting 0 16 Low
8 Enable Detecting 0 16 Low
9 Enable Detecting 0 16 Low
10 Enable Detecting 0 16 Low
11 Enable Detecting 0 16 Low
12 Enable Detecting 0 16 Low
13 Enable Detecting 0 16 Low
14 Enable Detecting 0 16 Low
15 Enable Detecting 0 16 Low
16 Enable Detecting 0 16 Low
17 Enable Detecting 0 16 Low
18 Enable Detecting 0 16 Low
19 Enable Detecting 0 16 Low
20 Enable Detecting 0 16 Low
21 Enable Detecting 0 16 Low
22 Enable Detecting 0 16 Low
23 Enable Detecting 0 16 Low
24 Enable Detecting 0 16 Low
25 Enable Detecting 0 16 Low
26 Enable Detecting 0 16 Low
27 Enable Detecting 0 16 Low
28 Enable Detecting 0 16 Low
29 Enable Detecting 0 16 Low
30 Enable Detecting 0 16 Low
31 Enable Detecting 0 16 Low
32 Enable Detecting 0 16 Low
33 Enable Detecting 0 16 Low
34 Enable Detecting 0 16 Low
35 Enable Detecting 0 16 Low
36 Enable Detecting 0 16 Low
37 Enable Detecting 0 16 Low
38 Enable Detecting 0 16 Low
39 Enable Detecting 0 16 Low
40 Enable Detecting 0 16 Low
41 Enable Detecting 0 16 Low
42 Enable Detecting 0 16 Low
43 Enable Detecting 0 16 Low
44 Enable Detecting 0 16 Low
45 Enable Detecting 0 16 Low
46 Enable Detecting 0 16 Low
47 Enable Detecting 0 16 Low
48 Enable Detecting 0 16 Low
ERS4500#show log
Type Time Idx Src Message
---- ----------------------------- ---- --- -------
I 2013-05-22 19:51:50 GMT+00:00 35801 PoE Module Reset
I 2013-05-22 19:53:01 GMT+00:00 35802 Trap: pethMainPowerUsageOnNotification
I 2013-05-22 19:53:01 GMT+00:00 35803 PoE Main Power Usage On 512 Watts
I 2013-05-22 19:55:25 GMT+00:00 35804 Trap: pethPsePortOnOffNotification
I 2013-05-22 19:55:25 GMT+00:00 35805 PoE Port Detection Status: Port 1 Status: Delivering Power
I 2013-05-22 19:56:35 GMT+00:00 35806 PoE Module Reset
I 2013-05-22 19:57:46 GMT+00:00 35807 Trap: pethMainPowerUsageOnNotification
I 2013-05-22 19:57:46 GMT+00:00 35808 PoE Main Power Usage On 512 Watts
I 2013-05-22 20:00:10 GMT+00:00 35809 Trap: pethPsePortOnOffNotification
I 2013-05-22 20:00:10 GMT+00:00 35810 PoE Port Detection Status: Port 1 Status: Delivering Power
I 2013-05-22 20:01:21 GMT+00:00 35811 PoE Module Reset
I 2013-05-22 20:02:32 GMT+00:00 35812 Trap: pethMainPowerUsageOnNotification
I 2013-05-22 20:02:32 GMT+00:00 35813 PoE Main Power Usage On 512 Watts
I 2013-05-22 20:04:55 GMT+00:00 35814 Trap: pethPsePortOnOffNotification
I 2013-05-22 20:04:55 GMT+00:00 35815 PoE Port Detection Status: Port 1 Status: Delivering Power
I 2013-05-22 20:06:06 GMT+00:00 35816 PoE Module Reset
I 2013-05-22 20:07:17 GMT+00:00 35817 Trap: pethMainPowerUsageOnNotification
I 2013-05-22 20:07:17 GMT+00:00 35818 PoE Main Power Usage On 512 Watts
I 2013-05-22 20:09:40 GMT+00:00 35819 Trap: pethPsePortOnOffNotification
I 2013-05-22 20:09:40 GMT+00:00 35820 PoE Port Detection Status: Port 1 Status: Delivering Power
I 2013-05-22 20:10:51 GMT+00:00 35821 PoE Module Reset
I 2013-05-22 20:12:02 GMT+00:00 35822 Trap: pethMainPowerUsageOnNotification
I 2013-05-22 20:12:02 GMT+00:00 35823 PoE Main Power Usage On 512 Watts
I 2013-05-22 20:14:25 GMT+00:00 35824 Trap: pethPsePortOnOffNotification
I 2013-05-22 20:14:25 GMT+00:00 35825 PoE Port Detection Status: Port 1 Status: Delivering Power
I 2013-05-22 20:15:36 GMT+00:00 35826 PoE Module Reset
I 2013-05-22 20:16:47 GMT+00:00 35827 Trap: pethMainPowerUsageOnNotification
I 2013-05-22 20:16:47 GMT+00:00 35828 PoE Main Power Usage On 512 Watts
I 2013-05-22 20:19:11 GMT+00:00 35829 Trap: pethPsePortOnOffNotification
I 2013-05-22 20:19:11 GMT+00:00 35830 PoE Port Detection Status: Port 1 Status: Delivering Power
I 2013-05-22 20:20:21 GMT+00:00 35831 PoE Module Reset
I 2013-05-22 20:21:32 GMT+00:00 35832 Trap: pethMainPowerUsageOnNotification
I 2013-05-22 20:21:32 GMT+00:00 35833 PoE Main Power Usage On 512 Watts
I 2013-05-22 20:23:56 GMT+00:00 35834 Trap: pethPsePortOnOffNotification
I 2013-05-22 20:23:56 GMT+00:00 35835 PoE Port Detection Status: Port 1 Status: Delivering Power
I 2013-05-22 20:25:06 GMT+00:00 35836 PoE Module Reset
I 2013-05-22 20:26:17 GMT+00:00 35837 Trap: pethMainPowerUsageOnNotification
I 2013-05-22 20:26:17 GMT+00:00 35838 PoE Main Power Usage On 512 Watts
I 2013-05-22 20:28:41 GMT+00:00 35839 Trap: pethPsePortOnOffNotification
I 2013-05-22 20:28:41 GMT+00:00 35840 PoE Port Detection Status: Port 1 Status: Delivering Power
I 2013-05-22 20:29:52 GMT+00:00 35841 PoE Module Reset
I 2013-05-22 20:31:02 GMT+00:00 35842 Trap: pethMainPowerUsageOnNotification
----More (q=Quit, space/return=Continue)----
So you can see from the above output that this is an ERS4548GT-PWR Rev:12 running software version 5.6.1.053. Along with this you can see that the switch cannot recognise the PoE module Firmware so it is displaying as "Unknown", the output of two of the PoE commands responding with "Internal Error" and finally the log is showing "PoE Module Reset" and associated logs almost every 5 minutes.
Avaya/Nortel obviously have seen this problem in the field before because over the years they have been releasing PoE firmware for their various switches in the event that the PoE firmware becomes corrupt. I found the following document on support.avaya.com, but had seen in earlier from the Nortel support website
ERS4500_PoE_Firmware_Readme_v1.pdf , 5.6.x
Looking for the PoE firmware on support.avaya.com, I was unable to find the correct PoE firmware for my model under the version 5.6.1 section, thinking it may be under the main release I checked the 5.6 section and there was still no PoE firmware available for my switch model. After finally checking in the 5.6.2 section I was able to find the following two files.
4500_PoE_400b15.img , 5.6.x
PoE Firmware for ERS 4500-PWR (hardware revision 12 or later: 4526T-PWR HW:12, 4550T-PWR HW:12, 4524GT-PWR HW:12, 4526GTX-PWR HW:12, 4548GT-PWR HW:12)
4500_PoE_400.img , 5.6.x
PoE Firmware for ERS 4500 (up to hardware revision 11: 4526T-PWR HW:11, 4550T-PWR HW:11, 4524GT-PWR HW:11, 4526GTX-PWR HW:11, 4548GT-PWR HW:11 )
MD5 Checksum:5ed16e97d5bef9019cb9c00243603539
As I had a hardware revision #12 ERS4548GT-PWR I downloaded "
4500_PoE_400b15.img" and copied it to my TFTP server.
I then attempted to download the file via TFTP to the switch
ERS4500#download address 10.0.0.1 poe_module_image 4500_PoE_400b15.img
% Invalid image
I had no idea why the switch was complaining that it was an "Invalid image", but I thought I would try the other file so I downloaded "
4500_PoE_400.img", copied it to my TFTP server and attempted to download this new file via TFTP to the switch
ERS4500#download address 10.0.0.1 poe_module_image 4500_PoE_400.img
% Invalid image
I was confused, I thought there may be a bug or the files I downloaded were corrupt, but further investigation was required so I logged a job with Avaya. Within an hour I was advised to perform the following:
- Upgrade the software version on the switch to version 5.6.2 or later.
- Download the PoE firmware 4500_PoE_400b15.img to the switch.
- Leave the switch on the current version or downgrade the switch back to 5.6.1.
I took heed to their device, upgraded the switch to 5.6.2 and voilĂ I was able to download the PoE firmware to the device, noting that the PoE firmware also requires a reboot.
Once the device was restarted there were no longer any PoE log errors, the switch could read the PoE module firmware and the PoE module commands were returning normal results.
Success!