Major Connectivity Issue in Mumbai – Investigation Ongoing

Major incident Mumbai Mumbai Peering Mumbai Network
2025-01-23 18:45 IST · 35 minutes

Updates

Update

A technical issue was identified with one of our key transport links between GPX1 and Tata LVSB data centers in Mumbai. Although the link appeared operational, it could not reliably transmit traffic due to a complex issue affecting its performance. OSPF worked because it uses basic connectivity checks through UDP packets, which were still able to pass over the link. MPLS, however, relies on more complex protocols that depend on TCP traffic, which the link couldn’t handle properly.

What Happened:
An unexpected optical issue caused the link to behave inconsistently, leading to disruptions in traffic flow. This issue required an in-depth investigation, as standard monitoring tools did not detect the underlying problem.

Actions Taken:
We rerouted traffic to ensure service stability and minimize impact.
Our vendor has been engaged for further analysis to identify the root cause and resolve the issue.
To strengthen future monitoring, we are implementing additional tools to detect and address similar challenges proactively.

Please note no interruptions are expected.

January 24, 2025 · 15:59 IST
Resolved

Please be informed that we isolated the issue which is narrowed to a transport link connecting 2 of our data centers. It is constantly flapping while traffic passing via it is breaking. We were able to shutdown the link and started investigating. Traffic is recovering, we will be testing the link to check the root cause and monitor it once resolved without pushing traffic over it. RFO will be posted once we have all the details.

January 23, 2025 · 19:58 IST
Issue

We are currently investigating a major issue affecting connectivity in Mumbai. Several devices are unreachable, including those in Tata LVSB and some other data centers. At this time, the cause of the issue is unclear, but it does not appear to be related to fiber cuts.
We understand that this may be impacting many customers, and we are actively working to identify the root cause and restore services as quickly as possible.
Further updates will be provided as soon as more information becomes available.
We appreciate your patience and understanding.

January 23, 2025 · 19:16 IST

← Back