Phones Unregistering, High Latency

[troab]

I'm having an odd problem at a customer I inherited

Here's the back ground:
-Call Center with about 30 or so phones that runs 24/7
-Total of 4 switchvoxes (all AA350's or 355's) we are only concerned with a problem on 1 of them
-Phones are a mix of Aastra and Polycom
-The VOIP provider Peer settings have 2 setup as SIP and 1 setup as IAX (I don't know why)

Every morning sometime between 4 and 5 am nearly all of the phones will unregister (or become unregistered - lose connection in any case)

To the best of my ability I have ruled out: a switch reboot, a PoE problem, a scheduled switchvox backup, a switchvox reboot or crash, and so on

When this happens the Switchvox is reachable and manageable vi HTTPS and the telephone cards (PRI and Analog) report no problems.

A packet capture during this period has not yet revealed anything more than the thousands of REGISTER packets you would expect to see as all of these phones swarm to get back o the system.

I have setup some SNMP monitoring to try to shed more light and have found these bits of data:
-The two peers using SIP drop off during this time period and report latency towards 10,000
-The IAX peer doesn't even flinch - it stays OK and latency stays under 100 showing no noticeable change in fluctuation or pattern
-The CPU spikes during the time period but not unreasonably so and it returns to normal as all the phone gradually get re-registered
-I can't get the SNMP OID for "number of reachable phones" to work - it always reports 0 - all the time

We are still researching and troubleshooting the network for possible problems but have yet to turn up anything.

Anybody have any bright ideas? I'm running out of them....

[hh2011]

When did it start?

Do you have to bounce the system to get them to register, or after they drop do they just register right back with the PBX?

[troab]

It's been going on for quite awhile. We have done reboots, and system updates - but during the actual "outage" there is no need to bounce the system, everything just comes back on its own

[hh2011]

Are the PBX and Phones all on the same switch?

[troab]

No there are several switches involved but I have as of yet been unable to trace or isolate the problem to particular switch or particular uplink

[hh2011]

Hmmmmmmmmmmmm....



How many end points does the server in question handle?

[troab]

In the neighborhood of 200...

The interesting thing is this occurrence happens not only when call volume is low and there only about 30 people in the building, but when it has been that way for 10-12 hours. 4am is about as far into the slow shift as you can get.

My theories are along the lines of:
a) Some scheduled process on the network that eats up bandwidth, but I have been unable to prove or disprove this sso far with packet captures
b) Something that the phones are doing on a daily cycle that overwhelms the Switchvox

I'm less sure about option A though now that I see that only 2 out of 3 peer connection are affected....

Only other oddity that w are attempting to track down is some rogue phone or local device that keeps sending failed SIP registration requests... it is on the local network, responds to pings, but no http/s... Hoping we can use some ARP tricks to track it down on Monday and remove. I doubt it's generating enough traffic to be related though.

[hh2011]

Are your switches centrally located in a stack?

Can you move the switchvox off the switch it lives on?

I agree, that bandwidth usage by something, is not kicking them off.....


What is the time between keep-alive packets on the switchvox to end point?

[troab]

The switches are somewhat of a nightmare with redundant paths and Spanning Tree and whatnot - not a design I would have selected as for all I know the problems could have something to do with it.

We have already relocated the Switchvox from its original connection (an 10/100 edge switch - ugh) to the core switch - as centrally located as it can get

Not sure exactly where I can figure out the keep alive packets question.

I did find something interesting though - the Aastras all seem to be defaulted to a RE-REGISTER interval of 15 - that seems incredibly short to me. I brought this up with Digium support but they didn't think much of it.

[brucexx]

What firmware are you running on the box. I remember there was a bug in 5.0.1 fixed in 5.0.2 (or later I am not sure exactly) that under specific circumstances dropped all registrations. There was something with registration attempts using wrong password that caused global problems including dropping all registrations.

[troab]

5.5.3
It was on 5.1.2 and Digium support said upgrade to fix it as it was a known issue: however you have the right of it - the known issue was pre-5.1.2 and was already fixed in 5.1.2 (I think it was 5.1.2, build 40062 in any case, the release notes don't give the 5.x number)

40062
released on 2011-10-21
Issues Resolved
SIP extensions no longer become unregistered under a specific, rare set of circumstances.

[brucexx]

so are you still experience the issue or this update fixed it ? I might misunderstand what is going on...

[troab]

Sorry, no the update did NOT fix the problem...

[brucexx]

What did the trace show just before they all lose registration ? Is there something the system sends to the phone or the phone tries to register but fails and after timing out becomes unregistered ?

I would setup a trace on the phone system and on one jack to which a phone is plugged in (use hub if you have or port mirroring if switch can do). After that compare the traces for this specific ip address and see if the phone is sending "register" to the phone system and is not receiving anything or the "register" does not get to the phone system for any reason - or maybe the phone system is sending something that makes the phone unregister.

You can use wireshark filter:
sip and ip.addr == 10.10.10.1 and ip.addr == 10.10.10.2
where 10.10.10.1 and 10.10.10.2 would be the phone and phone system ip addresses (just substitute your ips).

[mpickard247]

Did you ever find the problem with this??? I have the same issue. Phones are exhibiting high latency on local LAN 35-113ms! I am pulling my hair out and have exhausted almost every avenue.

[lprikockis]

I'd be interested in hearing of any solutions or theories on this sort of problem as well. I've had a similar issue-- though we have a mix of Polycom and old (ugh) Mitel phones, so for all I know, the Mitels could be part of the problem.

For what it's worth, we're running 5.1.2.

cleaning up some poorly configured network topology has reduced the problem quite a bit .. but we still see ocassions where bunches of phones de-register themselves, sometimes requiring a reboot (of the phone) to re-connect.

never been able to link the incidents to any consistent cause however.