Invalid reply digest error Avaya ERS8600 (part 1)

So you are getting an error "Received invalid reply digest from server" when you are trying to authenticate via RADIUS on an Avaya / Nortel ERS8600.


The error is seen as follows

Trying 10.0.0.1...
Connected to 10.0.0.1.
Escape character is '^]'.

Login: user
Password: **********
Received invalid reply digest from server Login:

So there are a few ways to resolve this issue, in this post we will look at two easy methods, a third will be explained in a later post.

Now depending on the software version running on the chassis you have two options, Using Java Device Manager (JDM) which is for pre 7.0 chassis' and Enterprise Device Manager (EDM) which is for software versions 7.0 and later.

Java Device Manager (JDM)

So open up your chassis in JDM and from the top menu select Security then Control Path... and finally RADIUS... as shown in the below image.

A new window "Radius" will open and you will see the "RADIUS Global" tab.  Now click on the "RADIUS Servers" tab and you will be shown the following.

 


You can now double click in each of the SecretKey fields next to the relevant RADIUS server Address to change the key.

Click the "Apply" button once you have changed the SecretKey to the new value and you are done.

You should now be able to logon to your ERS8600.

Trying 10.0.0.1...
Connected to 10.0.0.1.
Escape character is '^]'.

Login: user
Password: **********



ERS8600:5>

Enterprise Device Manager (EDM)

So open we web session to your chassis to use EDM and from the left menu select Security then Control Path... and finally RADIUS... as shown in the below image.

 A new window "Radius" will open and you will see the "RADIUS Global" tab.  Now click on the "RADIUS Servers" tab and you will be shown the following.

 As you can see I have removed the IP address' in the "Address" and "SourceIPAddr" column.  To change a "SecretKey" you must click on the RADIUS Server entry line that you wish to change.  **Note you can click on any field within the RADIUS Server entry line, it does not have to be the "Secret Key" column.

You are now able to access the "Change SecretKey" button that was previously greyed out.  Clicking on this button will bring up the following

Now change the RADIUS SecretKey for the server that you selected.

You should now be able to logon to your ERS8600.

Trying 10.0.0.1...
Connected to 10.0.0.1.
Escape character is '^]'.

Login: user
Password: **********




ERS8600:5>

 **NOTE the EDM method can only be performed if you are using local auth or if the RADIUS keys for the "web" interface (as noted in the "used-by web" ACLI syntax) are correct.  If you are unable to logon to EDM via RADIUS then you really need to set the RADIUS key via SNMP, which we will explore in a future post.

Success!

rxstats.txt file on AVAYA ERS8600

Have you ever noticed a file on your ERS8600's PCMCIA card called "rxstats.txt"?

Well this file is written by the CPU before the port is shutdown by the CP-Limit feature.

You can display the file contents locally on the system with the "more /pcmcia/rxstats.txt" command or you can copy the file off to a server via TFTP/FTP eg "copy /pcmcia/rxstats.txt 10.0.0.2:rxstats.txt" to read with your favorite text editor.



Here is an example output of a rxstats.txt file.

================================================================================
 [MON JUN 24 02:39:07 2013 UTC]: Shutting down port 3/48 for crossing CPP Rate Limit Threshold
 ================================================================================
 
Multicast: 1617 Packets Per Second
Broadcast: 0 Packets Per Second 
 

 Rx Statistics based on Queue Priority
 --------------------------------------------------------------------------------
 
Counters as dumped at MON JUN 24 02:39:05 2013 UTC
 
RxQ#0:         3139
RxQ#1:          222
RxQ#2:         3653
RxQ#3:          749
RxQ#4:          217
RxQ#5:            0

Counters as dumped at MON JUN 24 02:39:07 2013 UTC
 
RxQ#0:         3143
RxQ#1:          222
RxQ#2:         3659
RxQ#3:         3994
RxQ#4:          217
RxQ#5:            0

 Rx Statistics based on Packet Type
 --------------------------------------------------------------------------------

                   
Counters as dumped at MON JUN 24 02:39:05 2013 UTC
 UNKNOWN                   54
 BPDU                       6
 ARP                        5
 OSPF_HELLO               158
 OSPF_DB_DESC              58
 OSPF_LS_ACK             3584
 OSPF_LS_REQ               34
 OSPF_OTHER               744
 SONMP                    157
 TCP_OTHER                  6
 ICMP_ECHO_REPLY            5
 ICMP_ECHO_REQ             30
 VLACP                   3139
 
Counters as dumped at MON JUN 24 02:39:07 2013 UTC
 UNKNOWN                   54
 BPDU                       6
 ARP                        5
 OSPF_HELLO               158
 OSPF_DB_DESC              58
 OSPF_LS_ACK             3590

 OSPF_LS_REQ               34
 OSPF_OTHER              3989
 SONMP                    157
 TCP_OTHER                  6
 ICMP_ECHO_REPLY            5
 ICMP_ECHO_REQ             30
 VLACP                   3143

So you have this file which has dumped a heap of information on your lap and you are probably thinking "Wow, thanks Avaya for providing me with such an informative log" or sometime along those lines.  Well I had the same feeling when I first glanced at this file, expecting the answer to hit me in the face.  Actually when you read the above example carefully you can notice there are two different timestamps throughout the file the first is the time that CP-Limit actually trigged, which in this case is MON JUN 24 02:39:07 2013 UTC and the second is MON JUN 24 02:39:05 2013 UTC, which just happens to be 2 seconds before the event.

So with this new information in hand we can now see clearly what actually trigged the event, which is the "OSPF_OTHER" counter. Two seconds before the trigger this counter value was a measly 744 packets and two seconds later its 3989 packets, which is an increase of 3245 packets. In our case the CP-Limit multicast pps was set to 1500 packets and the event triggered at 1617 pps.  We can roughly calculate this value by dividing the increase of OSPF_OTHER packets by the duration, which is 3245 / 2 = 1622.5.  

While I have obtained a value of 1622 pps I am happy that CP-Limit has saved the CPU on my ERS8600, though I do wish that Avaya would polish this interface to make it easier for people fault finding these types of events.



Please drop me a comment if you have found this post useful or if it requires more information.

Avaya ERS8600 failed 8692 SF/CPU (core dump / crash)

Here is the console output of an 8692 SF/CPU module that would not boot correctly.

RMA action is required, I did not have time to check if a replacement RAM module would have fixed this problem.

The module was running as standby in a HA pair and originally crashed while I was copying files onto the device via TFTP off the master SF/CPU.

This output was captured after the module was removed and reinserted in the chassis.

options: 0x4
VX_DEALLOC_STACK

VxWorks Events
--------------
Events Pended on    : Not Pended
Received Events     : 0x0
Options             : N/A

Stack Trace:
10d1be4 vxTaskEntry    +68 : boot280 ()
10b41b0 boot280        +80 : openimage ()
10b4800 openimage      +11c: malloc ()
10ca0c8 malloc         +1c : memPartAlloc ()
10c8dd0 memPartAlloc   +18 : sWeedMemPartAlloc ()
10c8418 sWeedMemPartAlloc+2a8: 10c7db4 ()
10c7ff0 memPartOptionsSet+6b0: 10c79cc ()

Registers:

r0     = 29daf620   sp     =  4b9de50   r2     =        0   r3     =  39c8384
r4     =  520003c   r5     =   e00010   r6     =        1   r7     =        0
r8     = 2fdaf66c   r9     = 24b9e1cc   r10    =  600004c   r11    = 24b9e1cc
r12    = 20000044   r13    =        0   r14    =        0   r15    =        0
r16    =        0   r17    =        0   r18    =        0   r19    =        0
r20    =        0   r21    =        0   r22    =        0   r23    =        0
r24    =        0   r25    =   e00008   r26    =        1   r27    =        0
r28    =   e00010   r29    =        0   r30    =  520003c   r31    =  39c8384
msr    =     b030   lr     =  10c7ff0   ctr    =        0   pc     =  10c7a64
cr     = 40000024   xer    = 20000000


stack: base 0x4b9dfb0  end 0x4b89fb0  size 81904  high 2912   margin 78992




Copyright (c) 2010 Avaya, Inc.
CPU Slot 6:    PPC 745 Map B
Version:       5.1.3.1/003
Creation Time: Sep 17 2010, 10:07:37
Hardware Time: JUN 20 2013, 03:58:45 UTC
Memory Size:   0x10000000
Start Type:    warm
SILICONSYSTEMS VER2.00
open_file:can't open "/pcmcia/pcmboot.cfg" 0x380003
S_dosFsLib_FILE_NOT_FOUND
/flash/  - Volume is OK

Loaded boot configuration from file /flash/boot.cfg
Attaching network interface lo0... done.
Crash Detected, core-save for this slot ENABLED.

Press <Return> to stop core gather.
Attempting to open compressed file...
NOTICE - Core File Destination:
         Host: (LOCAL)
    Directory: /pcmcia
     Filename: core06_20130620035847.gz

bmain_copyCPCore: Failure opening compression file descriptor for "/pcmcia"
Press <Return> to stop auto-boot...
Loading /flash/p80a5131.img ... 12632884 to 43802836 (43802836)
Starting at 0x1000000...

SILICONSYSTEMS VER2.00
 Booting PMC280 Mezz HW. Please wait...
data access
Exception current instruction address: 0x010c7a64
Machine Status Register: 0x0000b030
Data Access Register: 0x24b9e1d0
Condition Register: 0x40000024
Data storage interrupt Register: 0x42000000
Task: 0x4b9dfb0 "boot280"

Base CPU took an exception

MPC106 cs:0x20800006 err1:0x72000000 err2:0x00000000 addr:0xA1000000 0x000000A1


options: 0x4
VX_DEALLOC_STACK

VxWorks Events
--------------
Events Pended on    : Not Pended
Received Events     : 0x0
Options             : N/A

Stack Trace:
10d1be4 vxTaskEntry    +68 : boot280 ()
10b41b0 boot280        +80 : openimage ()
10b4800 openimage      +11c: malloc ()
10ca0c8 malloc         +1c : memPartAlloc ()
10c8dd0 memPartAlloc   +18 : sWeedMemPartAlloc ()
10c8418 sWeedMemPartAlloc+2a8: 10c7db4 ()
10c7ff0 memPartOptionsSet+6b0: 10c79cc ()

Registers:

r0     = 29daf620   sp     =  4b9de50   r2     =        0   r3     =  39c8384
r4     =  520003c   r5     =   e00010   r6     =        1   r7     =        0
r8     = 2fdaf66c   r9     = 24b9e1cc   r10    =  600004c   r11    = 24b9e1cc
r12    = 20000044   r13    =        0   r14    =        0   r15    =        0
r16    =        0   r17    =        0   r18    =        0   r19    =        0
r20    =        0   r21    =        0   r22    =        0   r23    =        0
r24    =        0   r25    =   e00008   r26    =        1   r27    =        0
r28    =   e00010   r29    =        0   r30    =  520003c   r31    =  39c8384
msr    =     b030   lr     =  10c7ff0   ctr    =        0   pc     =  10c7a64
cr     = 40000024   xer    = 20000000


stack: base 0x4b9dfb0  end 0x4b89fb0  size 81904  high 2912   margin 78992




Copyright (c) 2010 Avaya, Inc.
CPU Slot 6:    PPC 745 Map B
Version:       5.1.3.1/003
Creation Time: Sep 17 2010, 10:07:37
Hardware Time: JUN 20 2013, 03:59:36 UTC
Memory Size:   0x10000000
Start Type:    warm
SILICONSYSTEMS VER2.00
open_file:can't open "/pcmcia/pcmboot.cfg" 0x380003
S_dosFsLib_FILE_NOT_FOUND
/flash/  - Volume is OK

Loaded boot configuration from file /flash/boot.cfg
Attaching network interface lo0... done.
Crash Detected, core-save for this slot ENABLED.

Press <Return> to stop core gather.
Attempting to open compressed file...
NOTICE - Core File Destination:
         Host: (LOCAL)
    Directory: /pcmcia
     Filename: core06_20130620035939.gz

bmain_copyCPCore: Failure opening compression file descriptor for "/pcmcia"
Press <Return> to stop auto-boot...
Loading /flash/p80a5131.img ... 12632884 to 43802836 (43802836)
Starting at 0x1000000...

SILICONSYSTEMS VER2.00
 Booting PMC280 Mezz HW. Please wait...
data access
Exception current instruction address: 0x010c7a64
Machine Status Register: 0x0000b030
Data Access Register: 0x24b9e1d0
Condition Register: 0x40000024
Data storage interrupt Register: 0x42000000
Task: 0x4b9dfb0 "boot280"

Base CPU took an exception

MPC106 cs:0x20800006 err1:0x72000000 err2:0x00000000 addr:0xA1000000 0x000000A1


options: 0x4
VX_DEALLOC_STACK

VxWorks Events
--------------
Events Pended on    : Not Pended
Received Events     : 0x0
Options             : N/A

Stack Trace:
10d1be4 vxTaskEntry    +68 : boot280 ()
10b41b0 boot280        +80 : openimage ()
10b4800 openimage      +11c: malloc ()
10ca0c8 malloc         +1c : memPartAlloc ()
10c8dd0 memPartAlloc   +18 : sWeedMemPartAlloc ()
10c8418 sWeedMemPartAlloc+2a8: 10c7db4 ()
10c7ff0 memPartOptionsSet+6b0: 10c79cc ()

Registers:

r0     = 29daf620   sp     =  4b9de50   r2     =        0   r3     =  39c8384
r4     =  520003c   r5     =   e00010   r6     =        1   r7     =        0
r8     = 2fdaf66c   r9     = 24b9e1cc   r10    =  600004c   r11    = 24b9e1cc
r12    = 20000044   r13    =        0   r14    =        0   r15    =        0
r16    =        0   r17    =        0   r18    =        0   r19    =        0
r20    =        0   r21    =        0   r22    =        0   r23    =        0
r24    =        0   r25    =   e00008   r26    =        1   r27    =        0
r28    =   e00010   r29    =        0   r30    =  520003c   r31    =  39c8384
msr    =     b030   lr     =  10c7ff0   ctr    =        0   pc     =  10c7a64
cr     = 40000024   xer    = 20000000


stack: base 0x4b9dfb0  end 0x4b89fb0  size 81904  high 2912   margin 78992




Copyright (c) 2010 Avaya, Inc.
CPU Slot 6:    PPC 745 Map B
Version:       5.1.3.1/003
Creation Time: Sep 17 2010, 10:07:37
Hardware Time: JUN 20 2013, 04:00:28 UTC
Memory Size:   0x10000000
Start Type:    warm
SILICONSYSTEMS VER2.00
open_file:can't open "/pcmcia/pcmboot.cfg" 0x380003
S_dosFsLib_FILE_NOT_FOUND
/flash/  - Volume is OK

Loaded boot configuration from file /flash/boot.cfg
Attaching network interface lo0... done.
Crash Detected, core-save for this slot ENABLED.

Press <Return> to stop core gather.
Attempting to open compressed file...
NOTICE - Core File Destination:
         Host: (LOCAL)
    Directory: /pcmcia
     Filename: core06_20130620040030.gz

bmain_copyCPCore: Failure opening compression file descriptor for "/pcmcia"
Press <Return> to stop auto-boot...
 3
monitor#

AVAYA ERS8600 8692SF with no console output

Just recently after attending a customer site I discovered a failed standby CPU/SF (in slot 6) in an  ERS8600 running 5.1.x software.  I initially discovered the fault remotely via Java Device Manager (JDM) when performing a configuration change on another connected switch.

When I physically inspected the ERS8600 I was confronted with the following:-

 

As I you can see, the CPU/SF in slot 6 is displaying 75% CPU utilization and all other LED's shown were solid.  I attempted to console into the card via the console port shown on the front and the card was not responding.  Not trusting my own equipment, which was borrowed anyway, I consoled into card 5 and was promptly presented with the login banner.  After checking the DCE/DTE console switch on both cards (which should be set to DCE if using a straight through serial cable), I consoled back into card 6, removed and reinserted the module, but still no console output.  I noticed that as soon as power was applied to the card, the CPU utilization graph on card 6 would light up to 75% and stay solid with no other activity.

I could tell this was not normal behavior as when the cards generally perform a Power On Self Test (POST) the LED's do have a sequence where they turn on and off and the CPU utilization graph would show some activity before console output would be displayed.  [I will try to capture this sequence in a future post.]

Now I have only seen a few 8692 CPU/SF cards fail and most of them have actually had faulty RAM.  So after replacing the 256MB RAM module with an Avaya 8691SF/8690SF/8190SF memory upgrade kit  (DS1411016) PDF "Installing a CPU Memory Upgrade ERS8600" that I had spare, I was able to get the module to boot.

Success!