Connectivity Issue Impacting Clients in Mumbai
Updates
Our vendor engineers have assisted in stabilizing the network. Services are currently operating normally, and the vendor is reviewing collected logs and working to reproduce the issue in order to identify a permanent fix.
We have also received guidance on mitigation steps that would allow us to avoid visible impact should the problem reoccur, ensuring our members will not experience disruptions.
Please note that Akamai AANP traffic (Premium CDN package) has not yet resumed. We are in contact with Akamai, and recovery of this traffic may take additional time.
We are currently working with our vendor on the ongoing network issue. As part of their live testing and troubleshooting, some members may experience intermittent outages. A few of our core routers based in different data centers are not behaving as expected, and vendor engineers are making changes in real time to isolate and resolve the root cause. At this stage what we can update is that the issue is a combination of control-plane label/forwarding updates during rapid route changes causing instability on core routers. The root cause is not clear because there are no live events in our network like fiber-cuts or flaps.
At this stage, service stability is not guaranteed while testing continues, but this work is necessary to reach a permanent fix. We will provide further updates as progress is made.
We sincerely apologize for the ongoing disruption and thank you for your patience.
The issue has reoccurred and appears to be related to our devices misbehaving after multiple fiber cuts.
Our teams are actively working on the resolution, and we will keep you updated on the progress.
We sincerely regret the inconvenience caused and appreciate your patience.
We have stabilized the network and confirmed that the data centers directly affected were GPX 2 (Equinix MB 2), Netmagic Chandivali, and certain members from GPX 1 (Equinix MB 1). In addition, several content providers were impacted, including Akamai AANP (serving Premium CDN traffic – awaiting cluster resumption by the Akamai team), Fastly, Netskope, Apple, and DigitalOcean.
Our investigation indicates the root cause may be a combination of faulty optics and core routers struggling to immediately update label tables during multiple route changes. We are working closely with equipment vendors and the provider of the affected link to gather full details and ensure resolution.
At present, services are stable and traffic is flowing normally. Our engineering team continues to actively monitor the situation.
We are currently experiencing an issue in our network impacting clients in Mumbai.
Our network team is actively working with the concerned teams to identify the root cause and implement corrective measures at the earliest possible time.
We sincerely apologize for the inconvenience this may cause and appreciate your patience and understanding while we work to restore services to normal.
← Back