Hmmmm,
What a bright sunny day it was. Monday When I came to office and I just sat on my seat.
The fear that I had in mind has came alive.. :(
It was the recent SIP DoS attacks that I faced on one of our servers that used temporary to upgrade other servers.
Well the culprits has exploit our subnet and started to play around our network. After few minutes I'm being in the seat our PBX (asterisk server) went dead. no ssh , no responding , As we suspect It was a DoS attack, hmmmmmmmmmmmmmm... sigh....
Had to give a reboot to the server to bring back things up and running again.
Got the attackers ip in /var/log/asterisk/full under NOTICE lines which showed failed register attempts.
as usual blocked from iptables.
iptables -A blocked -s x.x.x.x -j DROP
Hmmm. I realize these fu*king script kiddies won't stop from there. It looks attack was originating from a single ip (differ on each attempts), But they flood more than four servers on the network at the same time, so it 's a DDoS as I understood, typical compromised corporate LANs which NAT everything to a single IP.
These bot-nets are used to run heavy scans on asterisk servers all around the globe.
Attackers use a script written in python , the svwar.py script is the one which scan and exploit weak extensions in your asterisk box.
http://code.google.com/p/sipvicious/
Well.... I just thought to modify my asterisk monitoring script to mitigate this shit !
To get rid of this fu*ker.....
Monday was over, next day (Tuesday) they had strike again. I have worked with our sysadmins to setup proper firewall to block any sip packets (REGISTER requests) that will flood the server and in the mean time I complete the below code snippet (in perl) to block this damn as* holes... :)
##############################################
my @matching_lines =`grep 'Registration from .* failed for .*' /var/log/asterisk/messages /var/log/asterisk/messages.1`;
foreach $line (@matching_lines){
($s1,$s2) = split(/ failed for /,$line) ;
($s3,$s4) = split(/-/,$s2);
$s3 =~ s/'//g; # you get the attackers IP from here
$blacklist{$s3} +=1;
}
foreach $ip (keys %blacklist){
print "From $ip\t hit =".$blacklist{$ip}."\n";
if($blacklist{$ip} > 5000){
print "Blocking IP $ip\n";
$ipalert = `grep $ip /etc/blacklistips`;
if($ipalert== ""){
if(!($ip =~ m/^(subnets_that_u_wanna_exclude)/)){
`iptables -A blocked -s $ip -j DROP`;
`echo -e "Block the IP $ip Immediately\n SIP REGISTER HIT =$blacklist{$ip}\n"|mail -s " SIP Flood Alert" emailaddr `;
open(BLKLSTFILE, ">>/etc/blacklistips");
print BLKLSTFILE "$ip\n";
close(BLKLSTFILE);
}
}
}
}
##############################################
well the script worked like an angel :) charming one .. :D :D
By the time sysadmins had put nice set of iptables rules to block flooding hits.
iptables rule are as below,
we had chains calls asterisk,asterisk.reg and lognrej. this was append under the 'filter' table
--------------------------------------------------------------------------------------------------------
-A asterisk -j asterisk.reg -s 0.0.0.0/0 -m string --algo bm --string "REGISTER" --from 28 --to 100
-A asterisk -j ACCEPT -s 0.0.0.0/0
-A asterisk -j lognrej
-A asterisk.reg -j accept_request_from_you_network
-A asterisk.reg -j lognrej
Well ! theses helped to stopped those annoying dumb script-kiddies
I hope you guys get something or nothing ;) out of this for making your life bit easier in this kinda situations
:D :D
See ya folks .......
Wednesday, December 15, 2010
Monday, November 29, 2010
unrevealing all about a2billing...
Well It's being almost near to end of 2010...
sigh...........................
Thought to discuss an important topic regarding prepaid voip applications.
Well ... When I said 'prepaid' everyone will get the name in their mind 'a2billing' in an instant, if you have already struggle to setup an a2billing platform.
I thought to share the things that I got from my experience about a2billing and point out the important points in setting up an a2billing system.
a2billing platform is mostly used as a very famous open source calling card platform in voip.
I'm not going to explain how the basic a2biling installation will be done as it's too easy , just untar a compress file and move it to an apache website.
Well the most important things are how we going to configure the a2billing platform which will allow us to put it in an production environment.
Below are the most important entities in a2billing,
1.) rate card
2.) tariff plan
3.) customers ( actual calling card)
a) First thing we need to do after installing a2billing is to create a rate card entry via web interface.
under Rate Card -> create new Rate Card
b)After that we can get all the dialing prefixes from a telco operator who has the all country codes and prefixes.
c) Then we can generate a csv file to import rates to a2billing system.
Rate Card -> Import Rate Card
Here you can associate your newly created rate card to the rates that you are going to import.
.csv files should contain fields as below,
dialprefix;destination;rateinitial
after this you can have,
dialprefix;destination;rateinitial;buyrate;buyrateinitblock;initblock and if need other required fields as necessary.
d) Now we have ratecard,rates define. So next thing is to create a callplan (tariffplan)
Rate Card -> Create Call Plan
Once the Call Plan is created you can add the rate card that you have created for this Call Plan
e) Finally now We have created Rate Card,Rates,Call Plan. So It's time to generate some customers (calling cards)
Customers -> Generate Customers
Provide the Call Plan that you have created for this card set. So all cards generated will be in the same Call Plan.
Ahh haaa.. :)
Now a2biling engine is completely configured.
we have RateCard , Rates, Call Plan and Cards (Customers)
I'll tell what are the associated tables in database for each of above entities.
RateCard --> cc_tariffplan
Rates --> cc_ratecard
Call Plans -> cc_tariffgroup
Cards(Customers) -> cc_card
And cc_call will contain all cdr informations.
Well............ Well........ Well.... So that's the simplest setting up of a2billing platform.
Next Important thing is that the progressive rates that We could configure in a2billing to have much more complex rating plans that can be achieved easily ;)
Ohhhhhh. F**k It wasn't easy for me to figure out this part of a2billing without having doing some brain intensive thinking :D :D lol ...........
Ok The tricky part of a2billing is that you can setup progressive rate schemes. That means say you want to charge x/min rate for first 5 minutes and then for next 10 minutes you'd like to charge y/min rate and finally a z/min rate for the rest of the call.
EEeeeeeeeeeeeeeekkkkkkkkkkk.................
I think you didn't get it at the first I think. Lets take an example.
a2billing has four step progressive rating engine
stepa , stepb , stepc , and default rate.
each step has following rate attributes.
stepchargea -- like a flagfall when entering this cycle
chargea -- rate to be apply during this cycle
timechargea -- duration of this cycle
billingblocka -- billing increment
Ex:- Lets say We want to charge 0.2 per minute for first 5 minutes and then after 5 minutes we have a flagfall of 0.1 and then for next 10 minutes we have 0.4/min rate
and finally we charge 0.5 for the rest of the call.
So cycles for this call will be
0---------5 ------ 15 ----------- end of call
so we have
stepchargea -- 0
chargea -- 0.2
timechargea -- 300 ( 60x5)
billingblocka -- 60 ( per minute billing)
step b start after step a finishes that means on 6th minute of the call
stepchargeb -- 0.1
chargeb -- 0.4
timechargeb -- 600 ( 60x10)
billingblockc -- 60 ( per minute billing)
rateinitial -- 0.5
initblock -- 60
billingblock -- 60
So It goes like first 5 minutes stepa then from 6th to 15th minute stepb then for the rest rateinitial charge.
So thats all about progressive rates in a2billing.
And one final thing is that in the newest version we can authenticate caller based on callerid allowing us to provide prepaid service ( no PINs need to enter for authentication)
Also we can configure an a2billing system as a callback system . Allowing callers being callback to enter the PIN and proceed with making calls in this mode.
Hmmm..............
Hope this would be helpful to anyone who interested and struggling to figure out the mysteries of a2billing system.
It ain't no big deal once you figure out HOWTO ;)
See ya folks in Christmas next time .
Jingle bell jingle bell ..................................
lol :D
sigh...........................
Thought to discuss an important topic regarding prepaid voip applications.
Well ... When I said 'prepaid' everyone will get the name in their mind 'a2billing' in an instant, if you have already struggle to setup an a2billing platform.
I thought to share the things that I got from my experience about a2billing and point out the important points in setting up an a2billing system.
a2billing platform is mostly used as a very famous open source calling card platform in voip.
I'm not going to explain how the basic a2biling installation will be done as it's too easy , just untar a compress file and move it to an apache website.
Well the most important things are how we going to configure the a2billing platform which will allow us to put it in an production environment.
Below are the most important entities in a2billing,
1.) rate card
2.) tariff plan
3.) customers ( actual calling card)
a) First thing we need to do after installing a2billing is to create a rate card entry via web interface.
under Rate Card -> create new Rate Card
b)After that we can get all the dialing prefixes from a telco operator who has the all country codes and prefixes.
c) Then we can generate a csv file to import rates to a2billing system.
Rate Card -> Import Rate Card
Here you can associate your newly created rate card to the rates that you are going to import.
.csv files should contain fields as below,
dialprefix;destination;rateinitial
after this you can have,
dialprefix;destination;rateinitial;buyrate;buyrateinitblock;initblock and if need other required fields as necessary.
d) Now we have ratecard,rates define. So next thing is to create a callplan (tariffplan)
Rate Card -> Create Call Plan
Once the Call Plan is created you can add the rate card that you have created for this Call Plan
e) Finally now We have created Rate Card,Rates,Call Plan. So It's time to generate some customers (calling cards)
Customers -> Generate Customers
Provide the Call Plan that you have created for this card set. So all cards generated will be in the same Call Plan.
Ahh haaa.. :)
Now a2biling engine is completely configured.
we have RateCard , Rates, Call Plan and Cards (Customers)
I'll tell what are the associated tables in database for each of above entities.
RateCard --> cc_tariffplan
Rates --> cc_ratecard
Call Plans -> cc_tariffgroup
Cards(Customers) -> cc_card
And cc_call will contain all cdr informations.
Well............ Well........ Well.... So that's the simplest setting up of a2billing platform.
Next Important thing is that the progressive rates that We could configure in a2billing to have much more complex rating plans that can be achieved easily ;)
Ohhhhhh. F**k It wasn't easy for me to figure out this part of a2billing without having doing some brain intensive thinking :D :D lol ...........
Ok The tricky part of a2billing is that you can setup progressive rate schemes. That means say you want to charge x/min rate for first 5 minutes and then for next 10 minutes you'd like to charge y/min rate and finally a z/min rate for the rest of the call.
EEeeeeeeeeeeeeeekkkkkkkkkkk.................
I think you didn't get it at the first I think. Lets take an example.
a2billing has four step progressive rating engine
stepa , stepb , stepc , and default rate.
each step has following rate attributes.
stepchargea -- like a flagfall when entering this cycle
chargea -- rate to be apply during this cycle
timechargea -- duration of this cycle
billingblocka -- billing increment
Ex:- Lets say We want to charge 0.2 per minute for first 5 minutes and then after 5 minutes we have a flagfall of 0.1 and then for next 10 minutes we have 0.4/min rate
and finally we charge 0.5 for the rest of the call.
So cycles for this call will be
0---------5 ------ 15 ----------- end of call
so we have
stepchargea -- 0
chargea -- 0.2
timechargea -- 300 ( 60x5)
billingblocka -- 60 ( per minute billing)
step b start after step a finishes that means on 6th minute of the call
stepchargeb -- 0.1
chargeb -- 0.4
timechargeb -- 600 ( 60x10)
billingblockc -- 60 ( per minute billing)
rateinitial -- 0.5
initblock -- 60
billingblock -- 60
So It goes like first 5 minutes stepa then from 6th to 15th minute stepb then for the rest rateinitial charge.
So thats all about progressive rates in a2billing.
And one final thing is that in the newest version we can authenticate caller based on callerid allowing us to provide prepaid service ( no PINs need to enter for authentication)
Also we can configure an a2billing system as a callback system . Allowing callers being callback to enter the PIN and proceed with making calls in this mode.
Hmmm..............
Hope this would be helpful to anyone who interested and struggling to figure out the mysteries of a2billing system.
It ain't no big deal once you figure out HOWTO ;)
See ya folks in Christmas next time .
Jingle bell jingle bell ..................................
lol :D
Monday, November 15, 2010
Flooding ............... SIP :)
Hmmmmmmmmmm.....................
Last week being a tough one, I decide to write down something interesting that happened last Thursday.
I was just being back to seat after lunch. It's around 13.00 from SL time. When I unlocked my screen I had 100+ email notifications from one of the asterisk gateway that I setup temporarily till the original server upgrade to lenny :)
email notification stated that the server load average is more than 15 and I quickly logged in to the server and did a `top` :)
I saw that asterisk is consuming 100% CPU at the time where as It shouldn't be.
So Next thing I did was had a `tail` on `/var/log/asterisk/full` to see what's going on with asterisk .
Well I found the problem someone has run a stress tester against our server with SIP REGISTER requests. I have observer many register fail attempts in the asterisk log.
The guy must use sipp or sipsak kinda tool to generate this huge traffic.
asterisk couldn't cope the request rate hence ate the CPU at it will :)
So to shutdown this annoy attacker I simply add a DROP rule in iptables filter table as below,
'-A INPUT -j logndrop -s x.x.x.x'
and finally did a 'ifdown fw' , 'ifup fw' to add the rule to iptables.
Here we go CPU usage drops down to 15% and so do load average.
So I said "Bye bye flood ........." :)
Interesting, coincidence thing was we had flooding in colombo due to heavy rains on the same day :D :D lol
Later.......... We block the attacker subnet from the gateway router :)
It's really important to have basic firewall setup which allow only required connections to be made specially when your server is on the internet.
cause BAD people are always looking for troubles :p
Last week being a tough one, I decide to write down something interesting that happened last Thursday.
I was just being back to seat after lunch. It's around 13.00 from SL time. When I unlocked my screen I had 100+ email notifications from one of the asterisk gateway that I setup temporarily till the original server upgrade to lenny :)
email notification stated that the server load average is more than 15 and I quickly logged in to the server and did a `top` :)
I saw that asterisk is consuming 100% CPU at the time where as It shouldn't be.
So Next thing I did was had a `tail` on `/var/log/asterisk/full` to see what's going on with asterisk .
Well I found the problem someone has run a stress tester against our server with SIP REGISTER requests. I have observer many register fail attempts in the asterisk log.
The guy must use sipp or sipsak kinda tool to generate this huge traffic.
asterisk couldn't cope the request rate hence ate the CPU at it will :)
So to shutdown this annoy attacker I simply add a DROP rule in iptables filter table as below,
'-A INPUT -j logndrop -s x.x.x.x'
and finally did a 'ifdown fw' , 'ifup fw' to add the rule to iptables.
Here we go CPU usage drops down to 15% and so do load average.
So I said "Bye bye flood ........." :)
Interesting, coincidence thing was we had flooding in colombo due to heavy rains on the same day :D :D lol
Later.......... We block the attacker subnet from the gateway router :)
It's really important to have basic firewall setup which allow only required connections to be made specially when your server is on the internet.
cause BAD people are always looking for troubles :p
Sunday, October 10, 2010
DTMF nightmare ..... :)
Ok Just before telling about the server migration that I have completed this week there was another interesting thing happen to me and one of my colleague in sydney office as he got a problem with DTMF recognition on one of his IVR system.
we have asterisk based VPBX systems for our business customers so the problem was with one of these VPBX (virtual PBX)
concept of VPBX is very simple as you familiar with Virtual machines Virtual OSs and etc ...
Idea is to share one asterisk server instance among different people with different contexts.
It's really easily to visualize asterisk as the core is designed and more flexible to have visualization.
Ok that's enough about virtualizations .. bla bla bla......... :)
The incident was, that my friend was stuck in a place were he allowed people who dialed in to VPBX main IVR were able to dial the internal extension from the IVR, problem was that when he was tested from SIP - to SIP everything worked fine as it is, But he got the problem when he got incoming calls from the PSTN (more likely a SIP gateway)
he had a WaitExten after a Background(announcement) to allow people to key in internal extension (4 digit). But the problem was that after first DTMF received no other sequential DTMF tones were recognized by asterisk.
By the time, I was readying to migrate one of our server which we have hosted voicemail/ teleconferencing services to upgrade the server OS to lenny. It was a etch box , But now it is not :)
While I was preparing for this my buddy came and told about the problem he was having.
So I was also tested But couldn't figure out WTF :) is wrong with asterisk.
As he told he has tried all available dtmf methods which supported by asterisk as inband,rfc2833,info,auto
But non of these made no change to the problem.
Ok I also looked at the problem here'n there for the first two days as I had other things in my mind about the server migration.
On third day I decided to dive in to the deep of this DTMF hazel. It's becoming a nightmare for both of us since we couldn't rectify the issue.
Ok before going in to dive I'll explain what each dtmf method and how the things works in asterisk world :)
DTMF stands for Dual Tone Multi Frequency. Its kind a way how we send digits while we are in a telephone call.
In analog world we most of the time use `inband` that means without any special encoding DTMF tones were send as normal audio tones within a audio call.
with SIP we use RTP for media transmissions so for codecs like g711u/a which are loose-less , means that compression/decompression has almost zero effect on original data we can use inband as tones were not get destroyed by codec.
But codecs like g729a/b has to use different technique like rfc2833 or sip info in order to transmit DTMF reliably. Now lets see what the hell is this rfc28833 is all about.
rfc2833 is an out of band technique used to send DMTF tones by encoding them into special information. SIP INFO is also an outband technique where DMTF tones are transmitted via SIP INFO packets.
Ok Now lets get down to how we figure out this DTMF problem that we encountered with asterisk server.
Ok as the first thing I looked at sip.conf and found that there were multiple `peer` entries to the same ip which was our sip server ip address.
asterisk used `peer` or `friend` entities in SIP to authenticate sip calls. If you have a `peer` asterisk will try to match against the ip address of the incoming call from that `peer`. If you have a `friend` entity asterisk will authenticate against digest credentials (username/password)
Ok the problem was here that since there were multiple `peer` entries to the same ip so asterisk used the first in the list to match. Therefore changing other `peers` attributes like `dtmfmode` has no effect.
So I figured this out by doing a sip debug peer ip_address and observer asterisk CLI when an incoming call hits the dialplan.
Here we go asterisk pops out and saying that it found peer entry for ip with bla bla bla and match against the first peer entry in its list.
It's like follow,
[peer1]
ip=x.x.x.x
[peer2]
ip=x.x.x.x
[peer3]
ip=y.y.y.y
If you check both peer1 and peer2 has same ip address and when you do a sip show peers you'll most probably see peer2 as the first sip peer with ip x.x.x.x
So changes made to peer1 has no effect.
This was the problem. I changed the dtmf mode to rfc2833 in the peer that asterisk matches for an incoming call and it solved the problem. it is always better to used an outband technique to transmit DTMF tones as inband is very unreliable even though we use g711u/a.
So thats about the DTMF nightmare me and my partner faced lol :D :D
I have spend two more days on server migration work as the old server config was puzzled up by the previous guy who worked on it.
I had to spend many hours to figure out which are linked where with that server before doing the migration. Any how it was completed by Friday and old server was upgraded to lenny by sysadmins.
Hmmmmmm... too much for today so I'm gonna stop for now.
catch ya soooooooooooooooooooooooon...................
we have asterisk based VPBX systems for our business customers so the problem was with one of these VPBX (virtual PBX)
concept of VPBX is very simple as you familiar with Virtual machines Virtual OSs and etc ...
Idea is to share one asterisk server instance among different people with different contexts.
It's really easily to visualize asterisk as the core is designed and more flexible to have visualization.
Ok that's enough about virtualizations .. bla bla bla......... :)
The incident was, that my friend was stuck in a place were he allowed people who dialed in to VPBX main IVR were able to dial the internal extension from the IVR, problem was that when he was tested from SIP - to SIP everything worked fine as it is, But he got the problem when he got incoming calls from the PSTN (more likely a SIP gateway)
he had a WaitExten after a Background(announcement) to allow people to key in internal extension (4 digit). But the problem was that after first DTMF received no other sequential DTMF tones were recognized by asterisk.
By the time, I was readying to migrate one of our server which we have hosted voicemail/ teleconferencing services to upgrade the server OS to lenny. It was a etch box , But now it is not :)
While I was preparing for this my buddy came and told about the problem he was having.
So I was also tested But couldn't figure out WTF :) is wrong with asterisk.
As he told he has tried all available dtmf methods which supported by asterisk as inband,rfc2833,info,auto
But non of these made no change to the problem.
Ok I also looked at the problem here'n there for the first two days as I had other things in my mind about the server migration.
On third day I decided to dive in to the deep of this DTMF hazel. It's becoming a nightmare for both of us since we couldn't rectify the issue.
Ok before going in to dive I'll explain what each dtmf method and how the things works in asterisk world :)
DTMF stands for Dual Tone Multi Frequency. Its kind a way how we send digits while we are in a telephone call.
In analog world we most of the time use `inband` that means without any special encoding DTMF tones were send as normal audio tones within a audio call.
with SIP we use RTP for media transmissions so for codecs like g711u/a which are loose-less , means that compression/decompression has almost zero effect on original data we can use inband as tones were not get destroyed by codec.
But codecs like g729a/b has to use different technique like rfc2833 or sip info in order to transmit DTMF reliably. Now lets see what the hell is this rfc28833 is all about.
rfc2833 is an out of band technique used to send DMTF tones by encoding them into special information. SIP INFO is also an outband technique where DMTF tones are transmitted via SIP INFO packets.
Ok Now lets get down to how we figure out this DTMF problem that we encountered with asterisk server.
Ok as the first thing I looked at sip.conf and found that there were multiple `peer` entries to the same ip which was our sip server ip address.
asterisk used `peer` or `friend` entities in SIP to authenticate sip calls. If you have a `peer` asterisk will try to match against the ip address of the incoming call from that `peer`. If you have a `friend` entity asterisk will authenticate against digest credentials (username/password)
Ok the problem was here that since there were multiple `peer` entries to the same ip so asterisk used the first in the list to match. Therefore changing other `peers` attributes like `dtmfmode` has no effect.
So I figured this out by doing a sip debug peer ip_address and observer asterisk CLI when an incoming call hits the dialplan.
Here we go asterisk pops out and saying that it found peer entry for ip with bla bla bla and match against the first peer entry in its list.
It's like follow,
[peer1]
ip=x.x.x.x
[peer2]
ip=x.x.x.x
[peer3]
ip=y.y.y.y
If you check both peer1 and peer2 has same ip address and when you do a sip show peers you'll most probably see peer2 as the first sip peer with ip x.x.x.x
So changes made to peer1 has no effect.
This was the problem. I changed the dtmf mode to rfc2833 in the peer that asterisk matches for an incoming call and it solved the problem. it is always better to used an outband technique to transmit DTMF tones as inband is very unreliable even though we use g711u/a.
So thats about the DTMF nightmare me and my partner faced lol :D :D
I have spend two more days on server migration work as the old server config was puzzled up by the previous guy who worked on it.
I had to spend many hours to figure out which are linked where with that server before doing the migration. Any how it was completed by Friday and old server was upgraded to lenny by sysadmins.
Hmmmmmm... too much for today so I'm gonna stop for now.
catch ya soooooooooooooooooooooooon...................
Saturday, October 9, 2010
Sick of Sniffing.......
Well Its been about more than week since my last post , and haven't get a time last weekend to write down something in here.
Last week I had an interesting Idea on implementing a tool for our support engineers for them to troubleshoot voip related stuff.
The Idea was to provide what will happen to a SIP call which will originate / terminate from or to our network.
Before, they tend to come to me and hazzel and saying there's something wrong with this and that service, and All I do to figure out any call routing issue is to do a ngrep at the sip server and see WTF is going on :)
Being after sometime, I thought shouldn't I allow my colleges to see for them selves whats actually happening at our voip network for a call. :D
So I wrote a simple perl/cgi script which ngrep the interface for about 1 minute and showing the call trace in a higher level interface.
ngrep -pqrtW bylinestring_for_search
I had following challenges when I was working on this.
Problem
1.) There's no option to us to run ngrep for a specified time.
Solution
I used the timeout utility in linux to timeout the ngrep command after running it for about 1 minute.
Problem
2.) ngrep can only be run by root , but the script that I wrote is executed by apache means www-data in debian. So www-data can't open network interface card in promiscuous mode means can't execute ngrep.
Solution
I used `sudo` and allow www-data execute ngrep with root privilege.
"/etc/sudoers"
www-data ALL = NOPASSWD:/usr/bin/ngrep,/usr/bin/timeout
This allowed Apache to execute the commands that i used in my cgi script.
It was quite interesting and support guys get start use the tool before they come to me :)
I got some more stuff to share But feels sleepy now So I'll stop for the moment, I'll have my next post about a server migration that I did this week.. sigh.......................
Last week I had an interesting Idea on implementing a tool for our support engineers for them to troubleshoot voip related stuff.
The Idea was to provide what will happen to a SIP call which will originate / terminate from or to our network.
Before, they tend to come to me and hazzel and saying there's something wrong with this and that service, and All I do to figure out any call routing issue is to do a ngrep at the sip server and see WTF is going on :)
Being after sometime, I thought shouldn't I allow my colleges to see for them selves whats actually happening at our voip network for a call. :D
So I wrote a simple perl/cgi script which ngrep the interface for about 1 minute and showing the call trace in a higher level interface.
ngrep -pqrtW byline
I had following challenges when I was working on this.
Problem
1.) There's no option to us to run ngrep for a specified time.
Solution
I used the timeout utility in linux to timeout the ngrep command after running it for about 1 minute.
Problem
2.) ngrep can only be run by root , but the script that I wrote is executed by apache means www-data in debian. So www-data can't open network interface card in promiscuous mode means can't execute ngrep.
Solution
I used `sudo` and allow www-data execute ngrep with root privilege.
"/etc/sudoers"
www-data ALL = NOPASSWD:/usr/bin/ngrep,/usr/bin/timeout
This allowed Apache to execute the commands that i used in my cgi script.
It was quite interesting and support guys get start use the tool before they come to me :)
I got some more stuff to share But feels sleepy now So I'll stop for the moment, I'll have my next post about a server migration that I did this week.. sigh.......................
Saturday, September 25, 2010
Bug Fixing......
Well After tempnut2 was put in to live there were about 5 conferences being used so far...
sigh............
This week was very interesting as I got a bug to fix in our email to fax system and also with the conference server.
The one with conference server was straight forward and I knew about that before hand , But didn't get a time to fix it once and for all. It just a poor DTMF handling logic though
First I worked on the bug with the email to fax system where we allow people to send attachments in many formats like txt,html,rtf,png,jpg,bmp,pdf,tiff,doc,docx,ppt,xls
withing an email and send it out as a fax to the destination. All these formats were converted to pdf at processing before sending it out to fax server.
Problem was with txt and rtf formats where they were not actually tested before with the system.
When I started to investigate the problem with .rtf attachments after spending two three hours on a php script which is doing the email decoding and dispatch the attachment to the fax server What I realize was that any raw data of an attachment is pass through a php base64_decode() function which was causing .rtf files being corrupted.
The real problem was that it assumed that any attachment of an email is subjected to MIME encoding when email send to a server where as .rtf/.txt does not.
So the actual problem was that the .rtf files are not subject to any encoding as it send as plain text format in an email attachment.
.rtf documents have an html style syntax where everything in .rtf enclose within a {}.
So It just a matter of by passing the bas64_decode() function only for .rtf files....
ooooopzzzzzzzzzz... also for .txt files as well.
So All I have done for patch this is the below simple logic in email decoding script.
if($attachment== "rtf"){
$attachment = stripslashes($attachment); # get rid of \\
# some other string substitutions for specific email clients like outlook
}
else if($attachment== "rtf"){
$attachment= $attachment
}
else{
$attachment = base64_decode($attachment);
}
And here we go........ I applied the patch to production yesterday and all went good :)
Hmmm ......................................
about the problem with the conference server was that I have used wait_for_digit AGI function to capture the dtmf inputs on conference authentication in AGI script.
I used the stream_file function to play relevant voice prompts, problem was that If user start to input stuff while announcement is in progress I miss the first digit that user press
It's like below,
$agi->stream_file("enter-confroom");
$room = get_hash_terminate_input($agi);
So to fix it, I used the below solution which worked on fly :)
$intkey = $agi->stream_file("enter-confroom","0123456789"); # any digit will interrupt the voiceprompt [0-9]
$key = get_interrupt_key($intkey);
$room = get_hash_terminate_input($agi,$key);
get_interrupt_key($intkey) function handles the interrupt key well, as even if the user press a key or he just wait till voice prompts over doesn't matter :)
Hmmmm thats all about bug fixing............................ :D
I'm gonna stop for today........................................
sigh............
This week was very interesting as I got a bug to fix in our email to fax system and also with the conference server.
The one with conference server was straight forward and I knew about that before hand , But didn't get a time to fix it once and for all. It just a poor DTMF handling logic though
First I worked on the bug with the email to fax system where we allow people to send attachments in many formats like txt,html,rtf,png,jpg,bmp,pdf,tiff,doc,docx,ppt,xls
withing an email and send it out as a fax to the destination. All these formats were converted to pdf at processing before sending it out to fax server.
Problem was with txt and rtf formats where they were not actually tested before with the system.
When I started to investigate the problem with .rtf attachments after spending two three hours on a php script which is doing the email decoding and dispatch the attachment to the fax server What I realize was that any raw data of an attachment is pass through a php base64_decode() function which was causing .rtf files being corrupted.
The real problem was that it assumed that any attachment of an email is subjected to MIME encoding when email send to a server where as .rtf/.txt does not.
So the actual problem was that the .rtf files are not subject to any encoding as it send as plain text format in an email attachment.
.rtf documents have an html style syntax where everything in .rtf enclose within a {}.
So It just a matter of by passing the bas64_decode() function only for .rtf files....
ooooopzzzzzzzzzz... also for .txt files as well.
So All I have done for patch this is the below simple logic in email decoding script.
if($attachment== "rtf"){
$attachment = stripslashes($attachment); # get rid of \\
# some other string substitutions for specific email clients like outlook
}
else if($attachment== "rtf"){
$attachment= $attachment
}
else{
$attachment = base64_decode($attachment);
}
And here we go........ I applied the patch to production yesterday and all went good :)
Hmmm ......................................
about the problem with the conference server was that I have used wait_for_digit AGI function to capture the dtmf inputs on conference authentication in AGI script.
I used the stream_file function to play relevant voice prompts, problem was that If user start to input stuff while announcement is in progress I miss the first digit that user press
It's like below,
$agi->stream_file("enter-confroom");
$room = get_hash_terminate_input($agi);
So to fix it, I used the below solution which worked on fly :)
$intkey = $agi->stream_file("enter-confroom","0123456789"); # any digit will interrupt the voiceprompt [0-9]
$key = get_interrupt_key($intkey);
$room = get_hash_terminate_input($agi,$key);
get_interrupt_key($intkey) function handles the interrupt key well, as even if the user press a key or he just wait till voice prompts over doesn't matter :)
Hmmmm thats all about bug fixing............................ :D
I'm gonna stop for today........................................
Tuesday, September 14, 2010
Its' show time :)
Hmmm,
After spending on few busy weeks finally my latest project was put on live on tempnut2.
This is about voip teleconferencing.
Just like other projects , All I got for this project from my manager was,
"pasan , our current conference server is just a junk. we don't make any money out of it, make a new conference server which is more better".
So As the first task What I have done was quickly went through the current server. There were many problems with the system and used really bad programing techniques to address the requirements.
After first 3 days , I have completely sort out what need to be done and started to design a new architecture for a conference server.
I have used the below technologies to finally make it a finishing product at the end,
asterisk
php/mysql
proftpd
Mainly the conference server back-end engine was written using asterisk dial-plan , MeetMe application and php AGI scripts.
The tricky part for providing administrator menu was done by use of feature map facility in asterisk.
/etc/asterisk/features.conf
This allowed me to provide users (administrator) a menu driven system fire by specifed DTMF sequence.
On the first attempt , I tried to use MEETME_AGI_BACKGROUND and didn't work for me as I got sip channels for incoming calls which were cause no audio when we run a AGI script in background for MeetMe application.
So I come up with the feature map facility in asterisk to provide admin menu while in a conference.
After about 2 weeks time I have completed the conference engine, and thereafter it took about 2 and 1/2 months to put it to a production server as the it had many dependencies from various ppl.
all configuration related to conferencing is fully database driven and used odbc to integrate with asterisk via /etc/asterisk/extconfig.conf
specially our sysadmin manager is very strict on security aspects of the servers.
So I have to spend much time when setting up a ftp server for downloading conference recordings with proftpd.
Finally able to setup a secure server with the guidance from sysadmins.
Yesterday We put the server live and waiting for receiving new orders from customers :)
Many people have helped me on making this project a success.
I would like to thank all of them for their support.
specially web front-end applications for ordering and managing teleconferencing were done my other 2 php guys.
So that credit goes to them.
As I thought in the first It was not at all difficult to setup a stable interactive teleconference server using asterisk
So finally thanks Mark :) and *
After spending on few busy weeks finally my latest project was put on live on tempnut2.
This is about voip teleconferencing.
Just like other projects , All I got for this project from my manager was,
"pasan , our current conference server is just a junk. we don't make any money out of it, make a new conference server which is more better".
So As the first task What I have done was quickly went through the current server. There were many problems with the system and used really bad programing techniques to address the requirements.
After first 3 days , I have completely sort out what need to be done and started to design a new architecture for a conference server.
I have used the below technologies to finally make it a finishing product at the end,
asterisk
php/mysql
proftpd
Mainly the conference server back-end engine was written using asterisk dial-plan , MeetMe application and php AGI scripts.
The tricky part for providing administrator menu was done by use of feature map facility in asterisk.
/etc/asterisk/features.conf
This allowed me to provide users (administrator) a menu driven system fire by specifed DTMF sequence.
On the first attempt , I tried to use MEETME_AGI_BACKGROUND and didn't work for me as I got sip channels for incoming calls which were cause no audio when we run a AGI script in background for MeetMe application.
So I come up with the feature map facility in asterisk to provide admin menu while in a conference.
After about 2 weeks time I have completed the conference engine, and thereafter it took about 2 and 1/2 months to put it to a production server as the it had many dependencies from various ppl.
all configuration related to conferencing is fully database driven and used odbc to integrate with asterisk via /etc/asterisk/extconfig.conf
specially our sysadmin manager is very strict on security aspects of the servers.
So I have to spend much time when setting up a ftp server for downloading conference recordings with proftpd.
Finally able to setup a secure server with the guidance from sysadmins.
Yesterday We put the server live and waiting for receiving new orders from customers :)
Many people have helped me on making this project a success.
I would like to thank all of them for their support.
specially web front-end applications for ordering and managing teleconferencing were done my other 2 php guys.
So that credit goes to them.
As I thought in the first It was not at all difficult to setup a stable interactive teleconference server using asterisk
So finally thanks Mark :) and *
Saturday, August 28, 2010
sip1 Died ...............
Ok After a loooooooooooooooong time I got a time to have a look at my blog,
After I left my previous work place now it's being a quite a long time and many things happened during this time.
I joined exetel on 15th October last year.
Just like my previous carrier , in exetel I was focus more on VoIP+ linux related stuffs.
Had a wonderful time ever since I joined exetel , learn a lot and keen on working on new aspects of VoIP.
Ok there are so many stories to tell for this couple of months.
One of the most memorable incident was happened on 27th May 2010.
Now the story beginssssss.................. :)
I got about 10+ tree's to take care everyday on our VoIP server farm :)
sooo many tree's were there
first on I met was filbert then hazelnut
sooo whats the difference ???
then gradually mamoncillo , mongongo, pilinut , paradisenut etc :) among them i finally met sausagetree as well :D lol
It's sooo amazing about the philosophy behind our server naming......
sip1 is the heart of our voip network. basically it is the sip router which responsible of carry out all routing of sip call traffic in our network.
Ok When I join exetel the sip router which was there was a pretty much old guy who almost had various problems on scalability.
After couple of months pass , After I return to SL being in SYD for couple of weeks I was able to finish my first mission which is to deploy and upgrade the existing ACD system that we had.
After my first mission 2nd one was hit at me in a time that I never expected. On 27th ThursdAy
While I was walking towards my office in the morning I met 1 of my colleague and heard a bad news. All he say was "There's something wrong with voip ..."
sigh.....
I went to office within next couple of minutes and turned on my PC and check the status with my immediate manager and found that half of the voip network is not responding.
Since from the morning they have restart the sip server couple time , cause of sip1 being freeze after sometime.
Ok Now What to do ?????????
I got a call from my manager and what he said that we need to immediately swap server to a new one.
I got anacardium and All I had was the pretty old config from the sip1.
anacardium is equipped with latest sip server softwares.
And there are a hell of a lot of changes being made to the core of the software.
The worst thing was that my engagement was on next day. Being frank I got bit f..kd up for a moment :(
Then After couple minutes I started to concentrate on the task that has being given to me and it was a really urgent, important , difficult one though :'(
After struggling couple of hours I have completed setting up anacardium to replace sip1
sigh..........................................................................
Ok After I completed setting up anacardium all that was left was to swap the server ips and put anacardium to live. For that I had to wait till the sysadmins comes back to work from home.
Since it was really being late for the day I said to my manager that I gotta go. :(
He said "It's ok ..you can go.."
After that I came home and they have swap the server on mid night and anacardium was operating since then,
After I came back to work on monday , there was a small configuration miss on one of the modules that handle NAT traversing mechanism causing some people being not able to receive calls.
After couple of minutes I have figure out the cause and fixed it immediately.
Then for about two or three more days I had to work on fixing some stuff related to billing and etc.
last week sip1 shot dead, and we got tempnut2over it. I made tempnut2 a replica of anacardium,
sigh..........................................
After I left my previous work place now it's being a quite a long time and many things happened during this time.
I joined exetel on 15th October last year.
Just like my previous carrier , in exetel I was focus more on VoIP+ linux related stuffs.
Had a wonderful time ever since I joined exetel , learn a lot and keen on working on new aspects of VoIP.
Ok there are so many stories to tell for this couple of months.
One of the most memorable incident was happened on 27th May 2010.
Now the story beginssssss.................. :)
I got about 10+ tree's to take care everyday on our VoIP server farm :)
sooo many tree's were there
first on I met was filbert then hazelnut
sooo whats the difference ???
then gradually mamoncillo , mongongo, pilinut , paradisenut etc :) among them i finally met sausagetree as well :D lol
It's sooo amazing about the philosophy behind our server naming......
sip1 is the heart of our voip network. basically it is the sip router which responsible of carry out all routing of sip call traffic in our network.
Ok When I join exetel the sip router which was there was a pretty much old guy who almost had various problems on scalability.
After couple of months pass , After I return to SL being in SYD for couple of weeks I was able to finish my first mission which is to deploy and upgrade the existing ACD system that we had.
After my first mission 2nd one was hit at me in a time that I never expected. On 27th ThursdAy
While I was walking towards my office in the morning I met 1 of my colleague and heard a bad news. All he say was "There's something wrong with voip ..."
sigh.....
I went to office within next couple of minutes and turned on my PC and check the status with my immediate manager and found that half of the voip network is not responding.
Since from the morning they have restart the sip server couple time , cause of sip1 being freeze after sometime.
Ok Now What to do ?????????
I got a call from my manager and what he said that we need to immediately swap server to a new one.
I got anacardium and All I had was the pretty old config from the sip1.
anacardium is equipped with latest sip server softwares.
And there are a hell of a lot of changes being made to the core of the software.
The worst thing was that my engagement was on next day. Being frank I got bit f..kd up for a moment :(
Then After couple minutes I started to concentrate on the task that has being given to me and it was a really urgent, important , difficult one though :'(
After struggling couple of hours I have completed setting up anacardium to replace sip1
sigh..........................................................................
Ok After I completed setting up anacardium all that was left was to swap the server ips and put anacardium to live. For that I had to wait till the sysadmins comes back to work from home.
Since it was really being late for the day I said to my manager that I gotta go. :(
He said "It's ok ..you can go.."
After that I came home and they have swap the server on mid night and anacardium was operating since then,
After I came back to work on monday , there was a small configuration miss on one of the modules that handle NAT traversing mechanism causing some people being not able to receive calls.
After couple of minutes I have figure out the cause and fixed it immediately.
Then for about two or three more days I had to work on fixing some stuff related to billing and etc.
last week sip1 shot dead, and we got tempnut2over it. I made tempnut2 a replica of anacardium,
sigh..........................................
Subscribe to:
Posts (Atom)
