Debugging Voice.
Ok, we all want our CCIE Voice numbers, I imagine that’s why a heck of a lot of you are reading this blog ;). I want mine too believe me… Double CCIE is the goal this year, 4 by 2012. Will I make it? I guess I’ll see.
Moving on, in the CCIE Voice lab we have all already been warned: Expect tons of troubleshooting. Internetworkexpert’s coursework would have you believe that this is limited to simple stuff like phones in the wrong VLAN and stuff like that, IP Expert seem to take a much better approach that chances are it’s going to be more complicated.
Cisco themselves mentioned at Networkers in Las Vegas that the CCIE Voice lab will now have lots of focus on troubleshooting, up to 15 percent! They also specifically mentioned that the days of assuming your PSTN is going to be an E1 connection in the lab are long gone too (i.e. it may entirely be a SIP trunk or H.323 gatekeeper.)
With this in mind, I set out on a journey of discovering voice troubleshooting, what followed was learning some incredibly useful commands that I wanted to share with you all.
We are going to examine some SIP call flows, some H.323 call flows and some DTMF-relay in an effort to understand them all a bit better. I have never seen an absolutely comprehensive collection of all this info in one place. I truly hope this helps someone out there.
This tutorial assumes you have some basic SIP knowledge already, you should know about the different types of SIP messages (invite, OK, ACK) , the fact that its text-based, that it sends back error codes as 400’s or 500’s like a HTTP server (404 not found, 403 unauthorized, that sort of thing.) Its also important to note (and I think Mark snow who used to be at IP expert for clearing this up in his VOD) that the UAC and UAS terminology you always see in SIP makes it sound way more complicated than it is: A UAC is just the “Calling” party and the UAS the called party. Simple as that.
All testing assumes a SIP trunk between a CUCM and a CUCME. Our CUCME has IP address 1.1.1.1 and binds all its SIP media to this address. Our CUCM has the IP address 10.0.0.252. Finally we have a phone with the IP Address 192.168.10.21 registered to our CUCM.
The CUCM handset has the number 5008, the CUCME has the number 911
First call debug:
In this example I am going to make a simple call from my CUCM handset
G.711, our device “911” will call “5008”
Source address of CCME: 1.1.1.1
SIP Trunk Details: Standard SIP trunk in CUCM. Nothing special configured,
SIP CALL FLOW
current dial-peer:
dial-peer voice 9999 voip
destination-pattern 5...
session protocol sipv2
session target ipv4:10.0.0.252
codec g711ulaw
---- Digits Dialed on handset here ----
PeterCCIE18371#debug ccsip messages
SIP Call messages tracing is enabled
PeterCCIE18371#
Aug 24 14:10:45.879: //-1/xxxxxxxxxxxx/SIP/Msg/ccsipDisplayMsg:
Sent:
INVITE sip:5008@10.0.0.252:5060 SIP/2.0
This part might look fairly obvious. We are SENDING an invite to sip:5008@10.0.0.252, one of the key things I wanted to point out in this was the “Sent” command you see at the top, it can get extremely tricky when debugging SIP working out who sent what message, this should help clarify things a little.
Via: SIP/2.0/UDP 1.1.1.1:5060;branch=z9hG4bK3F1C54
The “Via” message helps us work out what IP Address we are sending out FROM, here you can see we have bound our SIP messages to send out as 1.1.1.1 (we used the bind command in our config previously) so this is a great way to work out where exactly your SIP messages are being sourced from.
Remote-Party-ID: "PSTN Phone"
From: "PSTN Phone"
Here we run into one of those things that makes it tricky to read SIP, you see here that even though we generated this message (the invite message) we have our phone (911) listed as a REMOTE PARTY? Well, to the server (the UAS, the CUCM that we are calling) we ARE the remote party. So again, important to pay attention to that bit right at the top that tells you if your sending or receiving this message. You will also note since we have only just sent this message we don’t have the display name for the number we are calling (5008) shown yet. We also have a “tag” which helps us identify messages related to this call.
Date: Tue, 24 Aug 2010 14:10:45 GMT
Call-ID: 3910C333-AEC011DF-8248B9C0-9A07DD7B@1.1.1.1
Supported: 100rel,timer,resource-priority,replaces,sdp-anat
Min-SE: 1800
The “Supported” keyword indicates the SIP features we support, in here we have timer refreshes and a few other bits and bobs. We also have a Min-SE which specifies our session interval and when it needs to be refreshed.
Cisco-Guid: 936399850-2931823071-2185476544-2584206715
User-Agent: Cisco-SIPGateway/IOS-12.x
Allow: INVITE, OPTIONS, BYE, CANCEL, ACK, PRACK, UPDATE, REFER, SUBSCRIBE, NOTIFY, INFO, REGISTER
Here we are saying “This is the type of device I am” (user-agent) (essentially, although its not always the case) and the allow shows all the SIP methods that we understand and support. Subscribe for example indicates we support presence.
CSeq: 101 INVITE
Max-Forwards: 70
Timestamp: 1282659045
Contact:
Expires: 180
Here we can see that we are specifying a max forward, this says how many times we will allow this call to be forwarded (how many proxies or gateways in the path) we also have a Contact field, this tells the UAS where to send the message back to when its ready to send return messages.
Allow-Events: telephone-event
Supported: precondition
Content-Type: application/sdp
Content-Disposition: session;handling=required
Content-Length: 206
This is one of the more important parts of the messages and one of the least understood. In SIP some messages will contain SDP information (session description protocol) this is the information SIP that is actually used to control things like codec negotiation, dtmf-relay and also what IP addresses media should actually be sent to (this will become important later.)
v=0
o=CiscoSystemsSIP-GW-UserAgent 391 9124 IN IP4 1.1.1.1
s=SIP Call
c=IN IP4 1.1.1.1
t=0 0
a=rtr
m=audio 17922 RTP/AVP 0 19
c=IN IP4 1.1.1.1
a=rtpmap:0 PCMU/8000
a=rtpmap:19 CN/8000
a=ptime:20
We have a lot of confusing looking attributes here so lets try and go through as many as we can, the “C” attribute you can see listed here tells the system who initiated the call, this will come to be important later. The M indicates what codec we want to use, and the attributes specify our dtmf-relays.
Aug 24 14:10:45.899: //-1/xxxxxxxxxxxx/SIP/Msg/ccsipDisplayMsg:
Received:
SIP/2.0 100 Trying
Date: Tue, 24 Aug 2010 14:10:45 GMT
From: "PSTN Phone"
Allow-Events: presence
Content-Length: 0
To:
Call-ID: 3910C333-AEC011DF-8248B9C0-9A07DD7B@1.1.1.1
Via: SIP/2.0/UDP 1.1.1.1:5060;branch=z9hG4bK3F1C54
CSeq: 101 INVITE
Here you see we “RECEIVED” a message back from the other end, the content length is 0, indicating there is no SDP in this message. We also see that it is sending 200 trying, which essentially means its received our request. This is an important message: If the gateway does not receive this message after sending a 101 invite within the “Trying” timeout, it will retry X number of times and if no response is received will inform the gateway. These timers can be changed under SIP-ua with:
Sip-ua
Timers trying
Retry
!
You might need to do this if you have two CUCM’s as I posted in a previous blogpost.
Aug 24 14:10:45.903: //-1/xxxxxxxxxxxx/SIP/Msg/ccsipDisplayMsg:
Received:
SIP/2.0 180 Ringing
Date: Tue, 24 Aug 2010 14:10:45 GMT
Call-Info:
Allow: INVITE, OPTIONS, INFO, BYE, CANCEL, ACK, PRACK, UPDATE, REFER, SUBSCRIBE, NOTIFY
From: "PSTN Phone"
Allow-Events: presence
P-Asserted-Identity:
Supported: Geolocation
Remote-Party-ID:
Content-Length: 0
To:
Contact:
Call-ID: 3910C333-AEC011DF-8248B9C0-9A07DD7B@1.1.1.1
Via: SIP/2.0/UDP 1.1.1.1:5060;branch=z9hG4bK3F1C54
CSeq: 101 INVITE
Now we are getting a little further, The other end has successfully found the number we have rung and is proceeding to ring on the handset. This message will be the last message you see until someone picks up the handset, this message times out OR they ignore you, redirect you or heck they just answer the call, whatever the case may be. We can see here at this point that the remote end has also let us know what capabilities (sip methods) this device supports. The content length is again 0 indicating that no SDP is sent with this message.
----- Call Answered Here --------
Aug 24 14:11:32.523: //-1/xxxxxxxxxxxx/SIP/Msg/ccsipDisplayMsg:
Received:
SIP/2.0 200 OK
Date: Tue, 24 Aug 2010 14:11:21 GMT
Call-Info:
Allow: INVITE, OPTIONS, INFO, BYE, CANCEL, ACK, PRACK, UPDATE, REFER, SUBSCRIBE, NOTIFY
From: "PSTN Phone"
Allow-Events: presence, kpml
This is what we have been waiting for! The handset has picked up so we have been sent a 200 OK message with important details for us to connect the media streams up with.
P-Asserted-Identity:
Supported: replaces
Supported: Geolocation
Remote-Party-ID:
Content-Length: 155
Require: timer
To:
Contact:
Content-Type: application/sdp
Here we can see we have a content length of 155 indicating there is an SDP message waiting and a content-type of application/sdp, we also see above a keyword Supported: replaces, this is one of the many RFC extensions to SIP and is to do with conferencing, these all become more important as we perform call-forwards and other activities.
Call-ID: 4E4D2A57-AEC011DF-824EB9C0-9A07DD7B@1.1.1.1
Via: SIP/2.0/UDP 1.1.1.1:5060;branch=z9hG4bK4019BC
CSeq: 101 INVITE
Session-Expires: 1800;refresher=uas
Here we see values for the session expire, 1800, indicating that a session refresh must occur during this time period. The responsibility of the refresher is also shown (the UAS)
v=0
o=CiscoSystemsCCM-SIP 2000 1 IN IP4 10.0.0.252
s=SIP Call
c=IN IP4 192.168.10.21
t=0 0
m=audio 22782 RTP/AVP 0
a=rtpmap:0 PCMU/8000
a=ptime:20
Here we see the SDP sent from the CUCM, as you can see we have less options in the RTP map, and we also have M for the media shown, you will also notice the c= IN IP4 showing the actual IP address of the handset itself. This is very important as this is where we will actually attempt to connect via RTP to. This address would change if you where using an MTP for example. Notice also the bit I have highlighted in red, this is the port number that the RTP will attempt to connect on. If you check the SDP we originally sent in the invite message this is our RTP port too
Aug 24 14:11:32.535: //-1/xxxxxxxxxxxx/SIP/Msg/ccsipDisplayMsg:
Sent:
ACK sip:5008@10.0.0.252:5060 SIP/2.0
Via: SIP/2.0/UDP 1.1.1.1:5060;branch=z9hG4bK411AF4
From: "PSTN Phone"
To:
Date: Tue, 24 Aug 2010 14:11:21 GMT
Call-ID: 4E4D2A57-AEC011DF-824EB9C0-9A07DD7B@1.1.1.1
Max-Forwards: 70
CSeq: 101 ACK
Allow-Events: telephone-event
Content-Length: 0
We send an acknowledgement indicating we received the message and the rtp stream is setup between our two endpoints:
PeterCCIE18371#show voip rtp connection detail
VoIP RTP active connections :
No. CallId dstCallId LocalRTP RmtRTP LocalIP RemoteIP
1 220 219 16950 22782 1.1.1.1 192.168.10.21
callId 220 (dir=2): called=5008 calling=911 redirect=
dest callId 219: called= calling=911 redirect=
1 context 6998C42C xmitFunc 60A2437C
Found 1 active RTP connections
As you can see the VOIP rtp connections match what we saw in the M output.
Finally here is an extremely useful command to confirm everything we have talked about here.
PeterCCIE18371#show sip-ua calls
SIP UAC CALL INFO
Call 1
SIP Call ID : A06260B3-AEC511DF-8260B9C0-9A07DD7B@1.1.1.1
State of the call : STATE_ACTIVE (7)
Substate of the call : SUBSTATE_NONE (0)
Calling Number : 911
Called Number : 5008
Bit Flags : 0xC04018 0x100 0x80
CC Call ID : 220
Source IP Address (Sig ): 1.1.1.1
Destn SIP Req Addr:Port : [10.0.0.252]:5060
Destn SIP Resp Addr:Port: [10.0.0.252]:5060
Destination Name : 10.0.0.252
Number of Media Streams : 1
Number of Active Streams: 1
RTP Fork Object : 0x0
Media Mode : flow-through
Media Stream 1
State of the stream : STREAM_ACTIVE
Stream Call ID : 220
Stream Type : voice-only (0)
Stream Media Addr Type : 1
Negotiated Codec : g711ulaw (160 bytes)
Codec Payload Type : 0
Negotiated Dtmf-relay : inband-voice
Dtmf-relay Payload Type : 0
Media Source IP Addr:Port: [1.1.1.1]:16950
Media Dest IP Addr:Port : [192.168.10.21]:21026
Options-Ping ENABLED:NO ACTIVE:NO
Number of SIP User Agent Client(UAC) calls: 1
SIP UAS CALL INFO
Number of SIP User Agent Server(UAS) calls: 0
Now in our next example we have forwarded the call onto another number.
Aug 24 14:55:12.043: //-1/xxxxxxxxxxxx/SIP/Msg/ccsipDisplayMsg:
Received:
INVITE sip:911@1.1.1.1:5060 SIP/2.0
Date: Tue, 24 Aug 2010 14:55:12 GMT
Call-Info:
Allow: INVITE, OPTIONS, INFO, BYE, CANCEL, ACK, PRACK, UPDATE, REFER, SUBSCRIBE, NOTIFY
From:
Allow-Events: presence, kpml
Supported: timer,resource-priority,replaces
Min-SE: 1800
Cisco-Guid: 2677761991-2932150751-2187049408-2584206715
Remote-Party-ID:
Content-Length: 0
User-Agent: Cisco-CUCM7.1
To: "PSTN Phone"
Contact:
Expires: 180
Call-ID: A06260B3-AEC511DF-8260B9C0-9A07DD7B@1.1.1.1
Via: SIP/2.0/UDP 10.0.0.252:5060;branch=z9hG4bK409c848218
CSeq: 107 INVITE
P-Preferred-Identity:
Session-Expires: 1800;refresher=uac
Max-Forwards: 70
Hidden away in all of this, is where we have actually been transfered to, the P-preffered-Identity: shown above shows where the forward is sent, Can you see where we have been transferred to?
If you said 5111, good on you!
Aug 24 14:55:12.063: //-1/xxxxxxxxxxxx/SIP/Msg/ccsipDisplayMsg:
Sent:
SIP/2.0 100 Trying
Via: SIP/2.0/UDP 10.0.0.252:5060;branch=z9hG4bK409c848218
From:
To: "PSTN Phone"
Date: Tue, 24 Aug 2010 14:55:12 GMT
Call-ID: A06260B3-AEC511DF-8260B9C0-9A07DD7B@1.1.1.1
CSeq: 107 INVITE
Allow-Events: telephone-event
Server: Cisco-SIPGateway/IOS-12.x
Content-Length: 0
Again we have the extremely confusing thing where the “From: “ listed is actually a separate number to us, it’s very difficult to read if you look at this to try and determine which side has sent the message, use the Sent: or received at the very top to work out who sent what message. Here we are sending a 107 Invite which is a special type of message.
Aug 24 14:55:12.063: //-1/xxxxxxxxxxxx/SIP/Msg/ccsipDisplayMsg:
Sent:
SIP/2.0 200 OK
Via: SIP/2.0/UDP 10.0.0.252:5060;branch=z9hG4bK409c848218
From:
To: "PSTN Phone"
Date: Tue, 24 Aug 2010 14:55:12 GMT
Call-ID: A06260B3-AEC511DF-8260B9C0-9A07DD7B@1.1.1.1
CSeq: 107 INVITE
Allow: INVITE, OPTIONS, BYE, CANCEL, ACK, PRACK, UPDATE, REFER, SUBSCRIBE, NOTIFY, INFO, REGISTER
Allow-Events: telephone-event
Remote-Party-ID: "PSTN Phone"
Contact:
Supported: replaces
Supported: sdp-anat
Server: Cisco-SIPGateway/IOS-12.x
Supported: precondition
Session-Expires: 1800;refresher=uac
Require: timer
Content-Type: application/sdp
Content-Length: 200
v=0
o=CiscoSystemsSIP-GW-UserAgent 2272 8359 IN IP4 1.1.1.1
s=SIP Call
c=IN IP4 1.1.1.1
t=0 0
m=audio 16950 RTP/AVP 0 19
c=IN IP4 1.1.1.1
a=rtpmap:0 PCMU/8000
a=rtpmap:19 CN/8000
a=ptime:20
Here we are sending the CUCM our IP address and RTP stream that we will be listening on, the CUCM is responsible for relaying this information to the transferred-to party.
Aug 24 14:55:12.271: //-1/xxxxxxxxxxxx/SIP/Msg/ccsipDisplayMsg:
Received:
ACK sip:911@1.1.1.1:5060 SIP/2.0
Date: Tue, 24 Aug 2010 14:55:12 GMT
From:
Allow-Events: presence, kpml
Content-Length: 155
To: "PSTN Phone"
Content-Type: application/sdp
Call-ID: A06260B3-AEC511DF-8260B9C0-9A07DD7B@1.1.1.1
Via: SIP/2.0/UDP 10.0.0.252:5060;branch=z9hG4bK40d3e3b89db
CSeq: 107 ACK
Max-Forwards: 70
v=0
o=CiscoSystemsCCM-SIP 2000 7 IN IP4 10.0.0.252
s=SIP Call
c=IN IP4 192.168.10.30
t=0 0
m=audio 64878 RTP/AVP 0
a=rtpmap:0 PCMU/8000
a=ptime:20
Here we received a message back telling us no problem and where we need to connect to with our RTP stream, can you work out where we will be connecting to?
Here is another hint if you can’t get it from the above:
PeterCCIE18371#show voip rtp connections
VoIP RTP active connections :
No. CallId dstCallId LocalRTP RmtRTP LocalIP RemoteIP
1 220 219 16950 64878 1.1.1.1 192.168.10.30
Found 1 active RTP connections
We can finally do our show sip-ua calls for more info, note that annoyingly, even though our display on our phone has updated to indicate we are calling 5111, the number shown in the sip-ua calls is still 5008
PeterCCIE18371#show sip-ua calls
SIP UAC CALL INFO
Call 1
SIP Call ID : A06260B3-AEC511DF-8260B9C0-9A07DD7B@1.1.1.1
State of the call : STATE_ACTIVE (7)
Substate of the call : SUBSTATE_NONE (0)
Calling Number : 911
Called Number : 5008
Bit Flags : 0xC04018 0x10300 0x80
CC Call ID : 220
Source IP Address (Sig ): 1.1.1.1
Destn SIP Req Addr:Port : [10.0.0.252]:5060
Destn SIP Resp Addr:Port: [10.0.0.252]:5060
Destination Name : 10.0.0.252
Number of Media Streams : 1
Number of Active Streams: 1
RTP Fork Object : 0x0
Media Mode : flow-through
Media Stream 1
State of the stream : STREAM_ACTIVE
Stream Call ID : 220
Stream Type : voice-only (0)
Stream Media Addr Type : 1
Negotiated Codec : g711ulaw (160 bytes)
Codec Payload Type : 0
Negotiated Dtmf-relay : inband-voice
Dtmf-relay Payload Type : 0
QoS ID : -1
Local QoS Strength : BestEffort
Negotiated QoS Strength : BestEffort
Negotiated QoS Direction : SendRecv
Local QoS Status : None
Media Source IP Addr:Port: [1.1.1.1]:16950
Media Dest IP Addr:Port : [192.168.10.30]:64878
Options-Ping ENABLED:NO ACTIVE:NO
Number of SIP User Agent Client(UAC) calls: 1
SIP UAS CALL INFO
Number of SIP User Agent Server(UAS) calls: 0
Super quickly, here is the output when we try and make a call to a number that is not actually registered to the CUCM:
Aug 24 15:12:01.679: //-1/xxxxxxxxxxxx/SIP/Msg/ccsipDisplayMsg:
Sent:
INVITE sip:5999@10.0.0.252:5060 SIP/2.0
Via: SIP/2.0/UDP 1.1.1.1:5060;branch=z9hG4bK4878F
Remote-Party-ID: "PSTN Phone"
From: "PSTN Phone"
To:
Date: Tue, 24 Aug 2010 15:12:01 GMT
Call-ID: C8034B17-AEC811DF-8266B9C0-9A07DD7B@1.1.1.1
Supported: 100rel,timer,resource-priority,replaces,sdp-anat
Min-SE: 1800
Cisco-Guid: 3343539084-2932347359-2187442624-2584206715
User-Agent: Cisco-SIPGateway/IOS-12.x
Allow: INVITE, OPTIONS, BYE, CANCEL, ACK, PRACK, UPDATE, REFER, SUBSCRIBE, NOTIFY, INFO, REGISTER
CSeq: 101 INVITE
Max-Forwards: 70
Timestamp: 1282662721
Contact:
Expires: 180
Allow-Events: telephone-event
Supported: precondition
Content-Type: application/sdp
Content-Disposition: session;handling=required
Content-Length: 206
v=0
o=CiscoSystemsSIP-GW-UserAgent 125 8459 IN IP4 1.1.1.1
s=SIP Call
c=IN IP4 1.1.1.1
t=0 0
a=rtr
m=audio 17352 RTP/AVP 0 19
c=IN IP4 1.1.1.1
a=rtpmap:0 PCMU/8000
a=rtpmap:19 CN/8000
a=ptime:20
We send a SIP message with all our details…
Aug 24 15:12:01.703: //-1/xxxxxxxxxxxx/SIP/Msg/ccsipDisplayMsg:
Received:
SIP/2.0 100 Trying
Date: Tue, 24 Aug 2010 15:12:01 GMT
From: "PSTN Phone"
Allow-Events: presence
Content-Length: 0
To:
Call-ID: C8034B17-AEC811DF-8266B9C0-9A07DD7B@1.1.1.1
Via: SIP/2.0/UDP 1.1.1.1:5060;branch=z9hG4bK4878F
CSeq: 101 INVITE
We receive a trying message back…
Aug 24 15:12:01.707: //-1/xxxxxxxxxxxx/SIP/Msg/ccsipDisplayMsg:
Received:
SIP/2.0 404 Not Found
Reason: Q.850;cause=1
Date: Tue, 24 Aug 2010 15:12:01 GMT
From: "PSTN Phone"
Allow-Events: presence
Content-Length: 0
To:
Call-ID: C8034B17-AEC811DF-8266B9C0-9A07DD7B@1.1.1.1
Via: SIP/2.0/UDP 1.1.1.1:5060;branch=z9hG4bK4878F
CSeq: 101 INVITE
404 not found!
Aug 24 15:12:01.707: //-1/xxxxxxxxxxxx/SIP/Msg/ccsipDisplayMsg:
Sent:
ACK sip:5999@10.0.0.252:5060 SIP/2.0
Via: SIP/2.0/UDP 1.1.1.1:5060;branch=z9hG4bK4878F
From: "PSTN Phone"
To:
Date: Tue, 24 Aug 2010 15:12:01 GMT
Call-ID: C8034B17-AEC811DF-8266B9C0-9A07DD7B@1.1.1.1
Max-Forwards: 70
CSeq: 101 ACK
Allow-Events: telephone-event
Content-Length: 0
We send back an acknowledgement of this.
There you have it, if you made it this far, well done! I am glad to see your dedication. Why don’t you try debugging with G.729, try with an MTP clicked on your CUCM trunk and you will see things like the IP address for the rtp stream changed, the RTP-map will change depending on what dtmf-relay you want to do and much more.
hey great post! thanks for sharing...
ReplyDeleteDear Peter,
ReplyDeletethis will be better if you can share the codec- negotiation information as well.
Please suggest any video material to get more into the SIP.
waiting for you reply.
thanks & Regard,
Sandeep
I think mimicking popular posts on other blogs is one of the best ways to get a good idea which will be popular.Such a lovely blog you have shared here with us. Really nice. Sydney Harbour Corporate Cruises provide Bucks party Sydney cruise and a private unique experience on Sydney harbour for all guests to enjoy.
ReplyDeleteMuch thanks to you such a great amount for setting aside the ideal opportunity for you by and by to share such a pleasant information. I really support to perusing your post.
ReplyDeleteBmw repair San Diego
What’s up, every time i used to check blog posts here in the early hours in the break of day, for the reason that i enjoy to gain knowledge of more and more.
ReplyDeleteHonda car on road price in Kalyan
If any one wants to be a successful blogger, after that he/she must read this paragraph, for the reason that it includes al} methods related to that.
ReplyDeleteCollision Repair Houston
If some one wants expert view on the topic of blogging and site-building after that i advise him/her to visit this website, Keep up the fastidious job. Classic car painting Rancho San Diego
ReplyDeleteextraordinary post, especially keen. I ask why another experts of the part genuinely never find this. You will longing to go on your making. I'm confident, you have a limitless perusers' establishment right now!
ReplyDeletehot woman dating
Okay you are true, YouTube is best video distribution site, as YouTube is a lightly no much streaming time rather than other web sites.Phone repair west island
ReplyDeleteMy family dependably say that I am squandering my time here at net, yet I know I am getting knowledge all the time by perusing such lovely posts.
ReplyDeleteRepair tablets add tag
This substance is safeguarded paying little personality to everybody's idea. By what system would I have the ability to discover more?
ReplyDeleteTablet fix
I think you are effectively right. I was satisfied starting late when I got one of our understudies say, "I like it here in light of the way that your not a pack of eggheads." We without a doubt enlist coaches who can add to a positive loving with the student.Nice to look at that such get prepared focuses are the spot to be, on the grounds that if all else fails, most understudies don't fit the course of action of get ready. Particularly now in the computerized time, schools require better ways to deal with oversee planning. A commitment of gratefulness is all together for the article.
ReplyDeleteRĂ©paration mobile
Nice Blog.Thank you for Sharing. We are leading erp software software solution providers in chennai.
ReplyDeleteerp providers in chennai | cloud erp providers in chennai
Great post, Thanks a lot for the kind of perfect topic I have not a lot of information about it but I have got an extra unique info in your unique post.
ReplyDeleteData Entry in India
Believe there are many more pleasurable opportunities ahead for individuals that looked at your site.
ReplyDeleteConstruction erp software in chennai
yeezy 500
ReplyDeletegolden goose
yeezy boost 350
goyard tote
vans outlet
moncler outlet
jordan shoes
michael kors handbags
golden goose outlet
nike epic react flyknit
I still dont get that forwarding part to 5111? How did you find that ? I dont see p- preferred identidy pointed to 5111???... can you explain it
ReplyDelete