|
SNAP DECISIONS
Fast action and cooperation between the government and private sector helped mitigate the commercial damage of a recent nationwide Internet disruption. The incident, however, has many thinking about redundancy and backup networks to cope with future outages.
BY KHALED EL-BARAMAWY AND RÉHAB EL-BAKRY
At 6 am on January 30, the Internet went down. It was not the usual outage, such as when a server overloads or telephone exchange goes down. The service interruption, the result of near simultaneous cuts to two submarine cables deep beneath the sea eight kilometers north of Alexandria, affected more than 70 percent of all Internet traffic in Egypt.
For most of the day, websites were inaccessible, information feeds dried up and millions of frustrated Egyptians trying to conduct their online business received Error 404 messages. It was as if Egypt, which in 10 short years has grown rapidly to embrace information technology, had been sent back to the technological Stone Age.
The two damaged submarine cables, identified as FLAG Europe-Asia and SEA-ME-WE-4, were the main data conduits between Europe and the Middle East and South Asia. Many countries in the region reported similar service disruptions, with call centers in India operating at 50 percent capacity. Egypt might have been knocked off the grid entirely, had it not been for a third submarine cable, SEA-ME-WE-3, which makes landfall near Suez. An older cable with a lower bandwidth capacity, SEA-ME-WE-3 was still operable and able to carry some of the load.
The service disruption affected almost every sector that relies on digital telecommunication. Call centers took a heavy blow as Voice over Internet Protocol (VoIP) calls routed through the two damaged cables suddenly fell silent. Airlines and hotels that use electronic reservation systems were unable to communicate. Financial institutions faced similar difficulties, though banks and brokers insist their core businesses were unaffected. And home users grew frustrated at being unable to browse the Web or check their e-mail.
Service was gradually restored as the Ministry of Communications & Information Technology (MCIT) coordinated the shift of Internet traffic to the intact third cable. Within 48 hours of the initial cut, most companies were reporting business as usual, though the Internet continued to experience bottlenecks during peak hours until the two cables were repaired nearly two weeks later. “The impact of this incident on telecommunications companies, the stock exchange, banks, hotels and other vital organizations peaked on the first day and subsided slowly over the next two days as they were able to resume working,” explains Emad El-Azhari, CEO of TE Data, the country’s largest ISP.
In all, the government estimates that damages from the first 24 hours of the Internet service disruption could reach as high as LE 50 million. The figure could be considerably higher when MCIT’s decision to make ISPs provide one month’s free service to their 5 million subscribers as compensation is factored in.
It remains yet unclear who is liable for the damages. An investigation into the cause of the damage to the two cables is still under way. The findings will determine whether or not the owners of the two cables will be required to pay restitution for the interrupted services.
Damage control
If anything, the Internet outage was a wake-up call. It highlighted the country’s dependency on three submarine cables, each barely the thickness of a finger. The fiber optic cables lie buried about 50 cm beneath the seabed where they are vulnerable to earthquakes, dragging anchors, fishing trawler equipment and – some fear – sabotage. [See sidebar]
Within hours of the Internet outage Minister of Communications and Information Technology Tarek Kamel had assembled a crisis response team to assess the impact of the service disruption and seek solutions to mitigate the damage to vital services and businesses. The team, headed by the minister himself, included representatives from MCIT, the National Telecommunication Regulatory Authority (NTRA), Telecom Egypt (TE), Internet service providers, and telecom and IT firms. Its first priority was to provide a quick, albeit temporary, solution to the outage.
Observers were impressed by how the government and private sector worked together to mitigate the impact of the Internet disruption. Telecom Egypt, which was less affected than many other businesses because most of its Internet and telecommunications traffic was carried on the only intact cable, SEA-ME-WE-3, agreed to transfer as much bandwidth as it could spare to Egyptian IT and telecom firms affected by the outage. Telecom Egypt subsidiary TE Data, the only ISP that did not collapse, provided a portion of its bandwidth to its competitors. “Since this is a crisis that affected all ISPs, all of them [had to] work together to bear the consequences and support [one another],” says Akil Beshir, chairman and CEO of Telecom Egypt.
The reallocation of bandwidth helped mitigate corporate losses due to the service disruption, but MCIT officials feared the diverted traffic could overload the only remaining cable. “The problem was you had the entire country’s [traffic] running through a 10-gigabyte [connection],” says Alaa El Shafie, executive vice president of Xceed calling center. “The other two cables are 80 gigabytes each. That was the main reason for the slowdown.”
With this in mind, Kamel made a preemptive decision to rent nine gigabytes of emergency capacity from France Telecom, Malaysia Telekom and Telecom Italia. The extra bandwidth was used to boost Egypt’s internet capacity until the damaged cables could be repaired.
The minister also appealed to the public for support during the crisis. His statement, carried by local media, urged citizens to refrain from downloading music and movies from the Internet until the cables were repaired. This would free up more of the limited capacity for businesses to use, he said.
In the meantime, Kamel’s crisis management team was working on parallel tracks to identify the cause of the Internet service disruption. Didier Lombard, chairman and CEO of France Telecom, one of 16 companies in the consortium that owns the SEA-ME-WE-4 cable, confirmed to Kamel that the company’s cable was cut and a ship would be dispatched to the site to repair it. Similar confirmation was received from UK-based FLAG Telecom, the owner of the second damaged cable.
Many people were frustrated to learn that it would take up to 14 days for the companies that own the two damaged cables to repair them, and that Internet service would be spotty until the repairs were completed. Amr Badawi, chairman of the NTRA, explains that the repair work was carried out as fast as was feasibly possible. “The international companies that own the two cables acted immediately, dispatching the repair ship Raymond Cruise to the site, but the bad weather conditions delayed its arrival for six days,” he says, pointing out that the repair work was particularly tedious because the cables are located at great depths under the surface of the sea.
Every bit counts
The first thought on many people’s mind when they learned of the extent of the Internet outage was “Is my money safe?” Rest assured, say banking experts, the severed cables posed no threat to banking deposits and investments because most financial institutions rely on private networks, and not the Internet, for secure transactions. They also have off-site backup data warehouses to ensure that account data is secure in case of a natural disaster.
Tamer Fahmy, chief information officer of Commercial International Bank (CIB), says his bank staff experienced some difficulty in communicating with the outside world, but their core business was unaffected by the Internet outage. “We realized that there was a problem with the Internet mostly because we couldn’t communicate via e-mail,” he says. “We are, by and large, a bank that depends heavily on [remote communications]. Very few of our memos, for instance, are actually circulated on paper. Most of our internal communications are done via e-mail. So this slowed everything down for us... but as for banking transactions within Egypt, they were unaffected. This includes regular transactions between the different branches of the bank, as well as for our clients using ATMs.”
He explains that most ATMs operate on private networks that are not vulnerable to Internet disruptions. “Domestic clients were unaffected because most banks either use a [secure direct connection] to link their ATMs to their systems or have several backup connections. That’s why most domestic clients were able to withdraw their money uninterrupted.”
Tourists, however, may have faced difficulties trying to withdraw cash from ATMs because a request must be sent from the machine to their home branch. “Depending on the network their bank uses, they may have been unable to access cash,” he adds.
Like many local banks, CIB’s network is connected through more than one ISP. If one ISP is down, another kicks in. Yet with two major telecommunications cables down, the bank’s Internet traffic slowed to a crawl.
GiroNil, a company that specializes in automated bill processing services, also reported a slowdown due to the Internet disruption, though again without any serious effect on core business activities. “We have our own network connecting us directly to banks,” explains Frans van Eersel, GiroNil’s managing director. “The whole idea of having so many lines [connecting] directly to banks and having several backup lines was initially designed so that if one line goes down, we would have another line fully functioning. We designed [the system] with sufficient backup and we had a contingency plan.”
He says that for the most part, GiroNil experienced only delays as companies that would traditionally transfer their payroll files to the GiroNil back office via e-mail were unable to do so due to the Internet outage. “Instead, we had to have the companies send the files to us manually either via CD or flash memory stick. This caused a slight delay mostly due to logistics, but at no point was our core business affected by this.”
One lesson learned, he continues, is that the company should have a detailed contingency plan outlining the manual transfer of files from its clients in case of another massive Internet outage.
Bears, bulls and bytes
The Cairo & Alexandria Stock Exchanges (CASE) weathered the Internet outage better than most expected, with trading volume showing little sign of dropping off. One reason for the low impact, suspects Noha Fahmy, assistant portfolio manager at Prime Holding, is that brokers and clients still, for the most part, favor low-tech communications. “For us, the main use of the Internet is to access information about prices and companies on the CASE,” she says. “We usually have several information feeds; two of them are local and two are international. So when the Internet was down, the feeds based outside of Egypt were down. But the local feeds were still working.”
According to Fahmy, portfolio managers and traders usually use the live feed to monitor prices and news about the different companies. When traders decide to either buy or sell stock, they call a broker on the phone and make the transaction manually. “Most of the time, brokerages will have a representative on the floor [of the CASE]. When we want to sell or buy stocks, we call our broker on the floor and the transaction is done internally. So the fact that we are not a very high-tech market, actually was in our favor.”
International investors usually conduct their transactions in the same way, monitoring the information online via the Internet, then calling a local broker to conduct the transaction on their behalf. “The worst that would have happened is that access to the information about companies was not as readily available. And again, even if we had lost all information feeds, they would have still been able to get news and prices from traders on the floor,” says Fahmy.
It was a different story for online traders, who account for about 12 percent of all transactions. With the Internet down, they were unable to access information, or make trades. But Fahmy says these traders represent only a small portion of all trades. “Perhaps if the outage had continued for longer than two trading days, we might have had a bit more of a slowdown, but, for the most part, we were unaffected and the volume of trading for the two days was well within the average of the month and the week.”
The average daily trading value for January was LE 1.7 billion, while CASE reported LE 1.5 billion in trades on January 30 and LE 1.4 billion the following day. “As the numbers clearly illustrate, the outage didn’t impact the volume of trade or the ability to conduct trades,” says Fahmy. “It also means that investor confidence was not affected at all by the outage.”
Hold that call
Egypt’s emergence as one of the fastest-growing offshore call center destinations would not have been possible without equally impressive growth in the country’s telecommunications sector. Half a dozen call centers with more than 5,000 seats handle domestic and international clients. While some sectors experienced only minor inconvenience as a result of the Internet disruption, for call centers it was akin to pulling out the plug. Most call centers in Egypt employ VoIP, which transmits voice calls via the Internet.
Hardest hit were call centers specialized in handling calls from North American and European clients, says Medhat Khalil, chairman of Raya Holding, the parent company of Raya Contact Center. “The amount of losses varied according to the nature of each center’s business and contracts. In our case, more than 60 percent of our operations cater to foreign markets,” he says. “What truly concerns [me] more than the fact that there will be compensation that has to be provided for [the drop in service], is the notorious reputation this might brand us.”
His sentiment is echoed by Adel Danish, chairman of Xceed, a 1,600-seat call center reduced to dead air when the Internet went down. He says the downtime put a sizeable dent in the company’s revenues, “but financial losses, no matter how high, are nowhere near the damage of the reputation of call centers. However, it is unlikely that [the outage] would have a deep impact in the long run, because what happened was out of everyone’s control, and was not due to a professional failure [on our part].”
Danish says the incident drew attention to the importance of having a contingency plan in the event of a disaster. Most international contracts stipulate that call centers must have hot sites. These backup facilities and off-site data storage warehouses ensure that data is backed up in case of natural disasters. Xceed and the larger call centers were equipped to handle earthquakes, fires and other disasters. But nobody anticipated that the country’s two main Internet lines would be rendered inoperable.
“We have a redundancy system in our business continuity plan so that if one of the submarine cables is out, then a second would function as backup,” explains Exceed’s El Shafie. “However, it was simply unforeseen that both the cables we depend on would be damaged.”
The situation would have been much worse had MCIT not managed to react as quickly as it had to the Internet outage, remarks Abdel Rahman El Sawy, a professor of communications engineering at Cairo University. “One of the [government’s] priorities was to provide assistance to call centers that serve international markets,” he says. “One solution... was to direct their international Internet traffic via alternative routes, south through Suez and through satellites.”
He explains that call centers, like ISPs, were allocated part of the capacity available to TE through the SEA-ME-WE-3 submarine cable. Xceed, for instance, was able to operate at 30-percent capacity after TE donated bandwidth until the main lines could be restored. Without this allocated bandwidth, the call centers would have been completely offline during the outage.
There are lessons learned that need to be taken into the consideration by all call centers in Egypt, says Khalil. “If we are serious about promoting a good reputation for these call center services [in Egypt], we must have what it takes and work on developing alternative solutions for such emergency cases,” he says.
Forward planning
The three submarine telecommunications cables that carry 95 percent of Egypt’s Internet traffic are self-redundant systems; half of their bandwidth is allocated as backup. Should any one line go down, the traffic can be rerouted and spread across the other two lines. But the Internet outage of January 30 illustrated the vulnerability of submarine cables, and had many in the industry considering whether additional backups are needed. “We were largely unaffected by the outages, but this experience illustrated that there is always a need for a backup system,” says CIB’s Fahmy. He says his company is currently considering various communication backups, such as alternate connection lines and satellite links.
Redundancy has been a hot topic since December 2006, when a major earthquake off the coast of Taiwan knocked out four major submarine cables. Mudslides that followed damaged nine other cables south of Taiwan, practically wiping the island off the telecommunications grid. It took almost 50 days for the cables to be repaired and caused enormous financial losses.
The repercussions of the Taiwan incident were impressed on the minds of MCIT’s decision-makers. Even before two damaged cables caused a massive Internet outage, the ministry was working to secure a deal for a fourth submarine cable linking Egypt to Europe. Ironically, the signing of a deal between Telecom Egypt and Alcatel-Lucent to construct this cable was scheduled for January 31, one day after the outage. The new fiber-optic cable is scheduled to be completed within 18 months at a cost of $125 million.
Orascom Telecom (OT) is building its own submarine cable to connect Egypt, Italy, Pakistan and the Gulf countries. “Introducing the OT [cable] in the first quarter of 2009 will contribute largely to handling such crises, and even preventing them from happening again,” says Karim Beshara, CEO of LINKdotNET, an OT subsidiary. “Work on this cable has been under way for several months, long before the Internet crisis [in January], and will cost $234 million.”
MCIT believes the two new submarine cables will help prevent a recurrence of the January 30 incident. “The installation of the Telecom Egypt-Alcatel-Lucent cable within 18 months, as well as the OT cable, which will be in service soon, will diversify the Internet resources we depend on,” Kamel explained.
The ministry is also looking to issue a license for a third new submarine telecom cable, bringing the country’s total to six. “We’ve already taken a decision to keep the extra Internet capacities that we rented, to be on the safe side in case of such incidents,” he added.
IT experts argue that measures such as redundancy and diversification of Internet resources can reduce the likelihood of communications failure, or minimize its damage, but it is impossible to anticipate every situation. The best the government and IT firms can do, is prepare for the worst.
|
A NET OF CONSPIRACY
Near simultaneous cuts in two of the three main Europe-Asia submarine broadband cables on January 30 caused massive service disruptions to Internet traffic to and from Egypt, the Middle East and India. Preliminary investigations into the cause of the two cuts are inconclusive, which has left room for a number of theories ranging from natural causes to sabotage.
The cuts in the two submarine cables were reported at a point eight kilometers north of Alexandria, where the two cables lie on the seafloor over a kilometer apart. As no seismic activity was reported in the area, initial reports surmised that a ship denied entry to Alexandria due to bad weather had altered its course, severing the two cables after dragging its anchor along the seabed.
The anchor drop theory hit a snag when the Ministry of Transport confirmed that there were no vessels in the vicinity of the two cables at the time of the outage, and the area is a designated no-go zone for ships on all navigation charts. As proof, the ministry provided satellite images and navigation reports of all ships in Egypt’s domestic waters for the period preceding the Internet outage.
With anchors all but ruled out, Egyptian officials began considering natural causes such as shifting current patterns. Minister of Communications and Information Technology Tarek Kamel pointed out that the cuts occurred during a period of unstable weather and that weather-related cuts have occurred elsewhere, such as in the territorial waters of Algeria and India.
But subsequent cuts of other submarine cables in the region gave rise to a number of alternative theories, ranging from terrorism to sabotage. On February 1, two days after the two cables were cut off the coast of Alexandria, a third cable was reported cut in the Persian Gulf, 56 kilometers north of Dubai. Two days later, a fourth submarine cable running between the UAE and Qatar was reported down. A fifth cut near Penang, Malaysia, interrupted service on a backdoor route to the Middle East.
Five cables in five days – it seemed too much to be written off as sheer coincidence. Even Kamel, who had initially ruled out sabotage, began to rethink his position. “The situation seems greater and graver than anticipated, and unprecedented. And although I’m not inclined to [entertain] conspiracy theories in such matters, the damage to [these cables] raises questions,” independent daily Al-Masry Al-Youm quoted the minister as saying.
Some suspected that a deliberate attempt was being made to isolate the Middle East, possibly in preparation for a US or Israeli attack on Iran’s nuclear facilities. To borrow a line from Star Wars: “A communication disruption can only mean one thing... invasion!”
Others suggested terrorism, or a false flag operation by a western government seeking a pretext for tighter security measures to protect these vital international communications channels. A naval presence and lucrative government contracts to private contractors specialized in security and surveillance might be on offer, they said.
While officials appear certain the third submarine cable was cut by a dropped anchor, and the fourth went offline due to an overloaded power supply, investigations into the cause of the two cable cuts near Alexandria, as well as near Penang, have been inconclusive. The UN’s International Telecommunication Union (ITU) says it has not ruled out sabotage.
|
Submit
your comment
Top
|
