Acme Packet Media (RTP) Troubleshooting

Who loves troubleshooting media issues? Not me. While some issues require a full packet capture, you can find out a number of things from a simple command. Like when you realize put a steering-pool in the wrong vlan…

I find myself using ‘show nat by-addr ‘ frequently. You do need to know the IP from the SDP c=line. A call must also be active when troubleshooting.

Basically this command gives you the VLAN info on the East / West sides from the call. It also provides the port used by the SDP (you can match this with your INVITE / 200OK ). It also displays the init_flow_guard, which in a normal scenario should be 4294967295.

The init_flow_guard is a timer that is set when the SBC receives a final 200OK of an INVITE transaction. If no media is sent, the timer begins to count down. As soon as the NAT table receives a single RTP packet, the ifg changes to 4294967295.

As always, mess around with it on your lab. See what you can discover on your network.
==========
sd1.# show nat by-addr 10.1.1.1

———————————
NAT table search address 801 :
Flow type: Fully qualified flow. Weight = 31
SA_flow_key : 10.1.1.1 SA_prefix : 32
DA_flow_key : 172.1.1.1 DA_prefix : 32
SP_flow_key : 0 SP_prefix : 0
DP_flow_key : 21136 DP_prefix : 15
VLAN_flow_key : 252
Protocol_flow_key : 17
Ingress_flow_key : 0
Ingress Slot : 0
Ingress Port : 0
NAT IP Flow Type : IPv4 to IPv4
XSA_data_entry : 72.X.X.X
XDA_data_entry : 216.X.X.X
XSP_data_entry : 24194
XDP_data_entry : 24120
Egress_data_entry : 1
Egress Slot : 1
Egress Port : 0
flow_action : 0X41
optional_data : 0
FPGA_handle : 0x00000000
assoc_FPGA_handle : 0x00000000
VLAN_data_entry : 193
host_table_index : 801
Switch ID : 0x00000002
average-rate : 0
weight : 0x1f
init_flow_guard : 4294967295
inact_flow_guard : 2
max_flow_guard : 86397
payload_type_2833 : 0
index_2833 : 0
pt_2833_egress : 0
qos_vq_enabled : 11
codec_type : 578155832
Call Recorder : 31304, Egress, Signal
HMU_handle : 578155832
sd1.ord1#

Session-Agent DNS Load Balancing with Acme Packet

I recently came across a scenario that required load balancing to an upstream SBC. This is easily accomplished by building a Session-Agent Group and setting the appropriate strategy, however I wanted to try a different path. I found that load balancing on the Acme Packet Net-Net series with SRV records is easy, if you follow a few basic steps.

1. Under the session-agent config, leave the IP address blank.
2. Set the port to “0”.
3. Add ‘dns-load-balance’ to the options under the SA.

session-router session-agent
    hostname enter.dns.name.here
    ip-address
    port 0
    state enabled
    options dns-load-balance

Acme Packet – Reject calls from a specific number

It’s been a while since my last update. Having another kid definitely limits the time to create a post, however she is sleeping right now. I’m sitting here with a Red Bull in one hand and a Secure CRT window open and ready to go.

Occasionally in the carrier world, your network might be blasted with a large number of calls from a dialer. While certain devices on the network might be able to handle the calls per second increase, others could experience an issue due to licensing. For instance, if you are running a BroadWorks Network Server and you are licensed for “X” transactions per second and this dialer puts you over the limit , you will start rejecting calls from valid subscribers.

Using the Acme Packet Net-Net series you have a few options for blocking these call attempts. One way is to route the calls to a fake realm / session agent and reject it from there. The issue with this method is that a SIP 503 is sent back up stream. If you have requested your upstream carrier to route advance upon receiving a 503, you’ve succeeded in migrating these call attempts to a different trunk on your network. This is most likely not what you are looking for. Instead you will need to have the SBC translate the 503 to a different response code.

Lets start with the mechanics of the response map. This actually very straight forward. you first supply a name for the response-map. Next you enter the transmitted code (recv-code : 503) and what you want it translated to (xmit-code : 404). You can also add a custom reason to the response code.


session-router
sip-response-map
name rejectCaller
entries
recv-code 503
xmit-code 404
reason "User not Found"
method
done
exit
done
exit


From here, you need a few more items in place (a realm, local-policy, an interface and a session-agent)

conf t
media-manager
realm-config
identifier rejectCaller
addr-prefix 0.0.0.0
network-interfaces M00:
average-rate-limit 16000
access-control-trust-level low
invalid-signal-threshold 1
maximum-signal-threshold 100
untrusted-signal-threshold 30
deny-period 60
done
exit

Next, you need a sip-interface:


sip-interface
state enabled
realm-id rejectCaller
description
sip-port
address 1.1.1.1
port 5060
transport-protocol UDP
tls-profile
allow-anonymous all
ims-aka-profile
done
exit
done
exit

…a Session Agent (notice the ‘local-response-code’ value is populated. This is where you enter the name of the response map that you have created)


session-agent
hostname 1.1.1.2
port 5060
state enabled
app-protocol SIP
transport-method UDP
realm-id rejectCaller
egress-realm-id
description "SA to apply response-map"
carriers
allow-next-hop-lp enabled
constraints disabled
max-sessions 1
max-inbound-sessions 1
max-outbound-sessions 1
max-burst-rate 1
max-inbound-burst-rate 1
max-outbound-burst-rate 1
max-sustain-rate 1
max-inbound-sustain-rate 1
max-outbound-sustain-rate 1
min-seizures 1
min-asr 1
time-to-resume 1
ttr-no-response 1
in-service-period 1
burst-rate-window 1
sustain-rate-window 1
local-response-map rejectCaller
done
exit

…A local-policy for routing of course :


local-policy
from-address +14801111111
to-address *
source-realm
description "Policy to send harassing calls to a fake SA"
activate-time N/A
deactivate-time N/A
state enabled
policy-priority none
policy-attribute
next-hop 1.1.1.2
realm rejectCaller
action none
app-protocol SIP
state enabled
methods INVITE
done
exit
done
exit

Add these config sections, perform a save / activate and you should be set.

Palladion – Scripting and time stamps.

I like to have Palladion execute scripts overnight and email the results in a CSV. The issue that I faced was that scripts require you to enter a start and end time. Logging in at 10pm to execute a post dial delay script was not an efficient use of my personal time.

I built the following functions to obtain the current time and then obtain the time from 24 hours ago. Then return these into the format that Palladion expects.  You can easily modify the “then()” function to look at a different time range.

import time
import datetime
from time import strptime

def now():
        # This fuction obtains the current time and returns it as a "datetime.datetime" object.
        now = time.gmtime()
        now2 = datetime.datetime(*now[:6])
        return now2

def then():
        # This function obtains the time from 24 hours ago and returns it as a "datetime.datetime" object.
        then1 = time.time()-86000
        then2 = time.gmtime(then1)
        then3 = datetime.datetime(*then2[:6])
        return then3

def run(facade,params): 
        #Script entry point. 
	start_ts = then()
	end_ts = now()

Palladion Extension Scripting

Palladion is a nice SIP analyzer from  IPTEGO. IPTEGO was recently purchased by Acmepacket… so they must be doing something right!

Anyways… here are a few “gotcha’s” that I have noticed while  working on extension scripts .

1. If you are trying to compare the time stamps between “Leg.setup_start_ts” and “Message.ts” it becomes tricky. SIP message timestamps from  Leg.getMessages are in “Y-m-d H:M:S” format. While Leg.setup_start_ts is in epoch format.You will need to build a function to convert one of these.  Check this blog  for an idea on how to convert datetime.datetime to epoch.

2. If you can’t seem to get a script to work, check in the “/tmp” directory and look for most recent folder. Review the file named “output” and you will see the traceback output.

3.  If you try to do something like :

f = open("/home/me/output.txt", "w")

print >>f, str

It will fail to open the file and the script will stop. If you change “f” to:

f = open("output1.txt", "w")

It should work, and the output will be found in /tmp/”most recent folder”. This is how I discovered the timestamp issue..

Speed up RTP packetloss troubleshooting with tshark read filters

I hate waiting for Wireshark to load a 20+meg capture file to review RTP streams. Once it loads you  browse  to “Telephony > RTP > Show All Streams”. Then you wait for it analyze the RTP streams…..

If you have a fair amount of memory it might not be too horrible of a wait, but it can always be quicker. Especially for those who are dowloading captures from a remote capture point. Often you download 5-6 files before you find one with any RTP loss or massive amounts of jitter.

Using “tshark” from the CLI ( Windows or *Nix) you can set a read filter and show the RTP stream analysis  in a few seconds.  The syntax is “tshark -r <filename > -qz rtp,streams“.  This process through the GUI takes about 3 minutes on my Windows Vista Laptop.  Running through the Windows CLI it takes under 10 seconds.

After running this you are presented with the same output seen through “Telephony > RTP > Show All Streams” (GUI).

Of course this method doesn’t allow listen to the G.711 streams like you can through the GUI. The GUI has the eavesdropping capabilities that allow you to hear issues that occurred before packetization (Stream shows no loss, however audio sounds choppy).  There is however a definite advantage to this method. Below a sample of the output :

[12:10:36][FSLab]$ tshark -r capture1.pcap -qz rtp,streams
========================= RTP Streams ========================
    Src IP addr  Port    Dest IP addr  Port       SSRC          Payload  Pkts         Lost   Max Delta(ms)  Max Jitter(ms) Mean Jitter(ms) Problems?
    192.168.3.38 23988    192.168.2.146 20000 0xFBD96BBC ITU-T G.711 PCMU   127     0 (0.0%)           20.07            0.03            0.02 X
   192.168.2.146 20000     192.168.3.38 23988 0x1D2E1E1A ITU-T G.711 PCMU   130     0 (0.0%)           23.48            0.43            0.07 X
    192.168.3.38 24016    192.168.2.146 20000 0x9F3C715C ITU-T G.711 PCMU   127     0 (0.0%)           20.16            0.03            0.02 X
   192.168.2.146 20000     192.168.3.38 24016 0x28781E1C ITU-T G.711 PCMU   130     0 (0.0%)           22.77            0.41            0.11 X
    192.168.3.38 23952    192.168.2.146 20000 0x173A4637 ITU-T G.711 PCMU   127     0 (0.0%)           20.07            0.03            0.02 X
   192.168.2.146 20000     192.168.3.38 23952 0x03C61E1E ITU-T G.711 PCMU   130     0 (0.0%)           20.32            0.11            0.08 X
    192.168.3.38 24012    192.168.2.146 20000 0x753488EF ITU-T G.711 PCMU   127     0 (0.0%)           20.09            0.03            0.02 X
    192.168.3.38 20372    192.168.2.146 20000 0xAB77B2B9 ITU-T G.711 PCMU   127     0 (0.0%)           20.19            0.04            0.02 X
   192.168.2.146 20000     192.168.3.38 20372 0x10021E47 ITU-T G.711 PCMU   130     0 (0.0%)          282.48           24.18            5.84 X