One of Pakistan’s main pipes to the global internet, the SEAMEWE-3 cable developed a fault on Monday at about 9pm Pakistan standard time. As of 11:30pm engineering at ITI informed me that the relevant parties were still trying to locate the fault, and there was no ETA on the fix. The fault has caused major data congestion on the few backup satellite uplinks Pakistan has to the outside world resulting in extremely slow to no connectivity in the entire region.
Update 12:38am: Seems like this is problem is not localized to Pakistan, it is also affecting India and Singapore’s bandwidth (and possibly everyone else who depends on SEAMEWE-3). ITI informs me that the SEAMEWE consortium is investigating this issue. Also – SEAMEWE-3 is Pakistan’s main pipe to the global internet.
Update 8:22am: Still no word on what the problem is or the ETA for the fix. All ITI says (or knows) is that work in on-going to fix the fault. They continue to work off backup satellites. Connection speed early morning (4am - 6am) was decent (60kbytes/sec on a 1mbs direct ITI link), presumably because of off peak hours. At the time of writing the 1mbs link is transferring at about 10kbytes/sec or about 10% of its capacity. It will probably get worse as the country wakes up and starts hitting the net.
Update 9:21am (Wednesday): Just spoke to an engineer in Islamabad (some guy from ITI Karachi gave me the number). The cable has not been repaired yet and Pakistan is still working off backup satellites. Apparently the repair crew is ready to fix cable fault, but has not received the green signal by all the countries who’s bandwidth will be effected by the repair operation (mostly countries in Asia and some in Europe). This story is also being covered on Slashdot as well as other sources .
Another interesting issue - Air Blue, a Pakistani airline carrier says that the fault has caused huge losses for them because they depend on their online reservation system for most of their business.
Corrected: It is SEAMEWE-3 cable that developed the fault, not SME-3.
Update 11:27pm (Wednesday): Just spoke to an engineer at ITI again. He confirmed what the local media is saying about repairs taking atleast 4-5 days and even as long as week. According to him, the repair crew has not reached the site yet, once they do they will need to physically trace the cable along the seabed and see what kind of damage has been done to it. Meanwhile, 3 backup E3 connections (one E3 connection is 33mbp/s I am told) have been deployed to cope with the bandwidth, though that doesn’t even meet half the bandwidth requirements of the Karachi at offpeak hours.
No planning, No backups!Goverment of Pakistan is with no planning, every body here in Pakistan is suffering from this problem, I’d got disconnected 9 times in last one hour.
ISPAK commentsISPAK’s general secretary Mr. V.A. Abdi (a great source of info) has just informed me that they’ve only just located the problem…
in his own words:
“as you know that the link is down since 8:45pm 27th June and is out till now. The fault, as reported by the GM ITR is a power-supply failure and the power to the fiber from Karachi end is not available. It has been discovered at about 12 Noon that the trouble is at a site 35 NM (42 miles) in the sea. A maintainance company by the name E marine have been asked to repair and their ship is starting from UAE now.”
gahh…
RE: ISPAK commentsI pity the call centers in Pakistan. Wonder how they are coping up with the traffic?
Fortunately the problem is no more than ~40 miles out.
RE: ISPAK commentsShoot… I’d be wondering how they were coping with the lack of traffic…
Fixed!I dont have the confirmtation from other ppl but in my office its working fine now. Getting 60kbps download speed!
RE: Fixed!I don’t see this in Karachi. Connectivity is still very shaky, though it has gotten better than yesterday. I think they setup some more backup bandwidth through satellite connections.
ptcl’s monopolyPreviously ISPs in Pakistan did have redundant satelite connections which they used for 2-way bandwidth. When the GOP and PTCL invested money into the Pakistan Internet Exchange (PIE) they were dismayed that ISPs were not using the exchange. So, with the help of the Government of Pakistan, PTCL enforced a law which stated that ISPs must use PIE for their upload stream and that they can only use their satelite connections for their downloading stream. This law, combined with PTCL’s overwhelming reliance on the SEAMEWE cable makes Pakistan’s entire internet connection funnel down to a single point of failure. Smart move by PTCL, eh?
The ISPAK (Internet Service Providers of Pakistan) made a huge row about the single point of failure about a year back, if I had the bandwidth, I’d link the story here.
frustrationsHaving just moved back from NYC and transfering part of our business operations to Pakistan, I missed my first net teleconference with my partners overseas thanks to the outage. The whole point of the internet is to provide alternative routes for data exchange rather than burden a single pipe. Half the phones at my new office are as it is dead (thanks to the recent strike) and now the internet outage really rubs the salt.
atleast the ISP has their backup bandwidth in place (be it at a crawl)
RE: frustrationsYou might want to look into solutions like ZakNet but they don’t come cheap.
Pay Yo Dam Bill!Maybe if you paid yo dam bill you wouldn’t be looking at a blank screen!
RE: frustrationsMy office has a 1mbs decidcated connection, it seems to be working normaly now (since 3am Friday) after some PTCL folks noted down our IP address. They are now prioritizing the bandwidth, if you can get them to prioritize your packets you should be good. They will need your static ip address for that.
Pathetic planningI found this to be amazing, that one line albeit the main line to the world goes down, and consequently brings down the countries net connection. Shouldn’t you guys have sufficient backup, which means that even if you have problems like this, it should affect the end user too much i.e he/she can still connect to the net without too much problem.
Instead of this debacle!
I wonder how this willI wonder how this will affect foreign investors who setup call centres. And how hard the airline and finance industries got hit. Those two heavily rely on the internet. I’m sure the stakeholders involved have taken a hit within millions of dollars.
RE: I wonder how this willPretty badly. Says alot about the country’s infrastructure when a single cable fault can take your business out for more than a week.
Well, on the very tiny upside of all this; at least they are learning some good leasons: always keep backups, prioritize bandwidth to businesses when an emergency hits aur subh kuch allah pay nah choro. They are also planning to invest about $40 million to lay down another backup cable sometime october of this year. Hopefully this investment can also increase the total bandwidth available to the country.
RE: I wonder how this willinteresting you bring this up since i’m a US based entrepreneur myself. the last year or so i’ve been advocating strongly to our investors to develop a presence (and a solid one at that) in Pakistan because of excellent cost savings compared to our Indian operations and rapidly developing technological infrastructure. Needless to mention that the recent turn of events have been a very big slap in the face for me personally, and investors have definitely lost a significant amount of confidence. The last week or so, I hate the fact that I’ve had to make reparations with potentially false promises that things will get better.
My 2 cents.