Call completion and voice services are experiencing issues
Incident Report for Cytracom
Postmortem

Summary of the Cytracom voice service disruption: October 31, 2019

On Thursday, October 31, we experienced a network issue that caused our end-users to lose the ability to make and receive phone calls. Additionally, access to the Cytracom platform via the web portal was disabled to prevent users from making configuration changes that would not have been effective during this event. We apologize for the disruption and wanted to provide an explanation.

What caused this issue?

We learned that there were several fiber cuts in the Dallas area that caused a loss of connectivity. The network instability caused by these cuts caused our failover between data centers to not work as intended. Put simply, the failover should have occurred to our secondary facility (KWD) once identified and we should have manually intervened much quicker than we did.

Timeline of events:

2:29PM CDT - Call completion disrupted
3:15PM CDT - Call completion partially restored
3:19PM CDT - Call completion interrupted due to an automated attempt to fail-back to primary
3:22PM CDT - PBX portal administratively shutdown as changes were not taking effect
3:59PM CDT - Call completion restored for all customers
4:15PM CDT - PBX portal administratively restored for customer and partner changes

What are the next steps?

Our engineering team is actively working to address this issue so it does not happen again. A full audit of systems and processes involved is underway and we will be providing further updates to our partners. If you have not done so already, we highly recommend you take a brief moment to review our Help Center article regarding QoS and how to configure all Cytracom subnets to ensure that in the event of a fail-over call quality is not compromised. You can find that information here under “Registration and Nodes”: Firewall best practices

Conclusion:

At this time, call completion remains stable and all Cytracom systems are operating normally. If you are still experiencing any difficulties, please contact our support desk and they will be happy to work with you to resolve any issues: 877-411-2987 or support@cytracom.com.

Posted Nov 01, 2019 - 10:09 CDT

Resolved
This incident has been resolved.
Posted Nov 01, 2019 - 10:04 CDT
Monitoring
Voice traffic has been restored as of 3:59 PM CDT. The dfw2 facility in Dallas has acknowledged the issue is related to several fiber cuts in Dallas, TX. We are awaiting restoration of the broken link, at which point we will restore services back to complete via their normal path. We are continuing to monitor the situation and will continue to provide updates.
Posted Oct 31, 2019 - 16:32 CDT
Identified
We have confirmed an upstream issue with a colocation provider. The issue is widespread, and affecting all facility customers. We are working closely with the colocation provider and will provide updates as they become available.
Posted Oct 31, 2019 - 15:12 CDT
Update
We have confirmed an upstream issue with a colocation provider. The issue is widespread, and affecting all facility customers. We are working closely with the colocation provider and will provide updates as they become available.
Posted Oct 31, 2019 - 15:12 CDT
Investigating
We have confirmed an upstream issue with a colocation provider. The issue is widespread, and affecting all facility customers. We are working closely with the colocation provider and will provide updates as they become available.
Posted Oct 31, 2019 - 14:45 CDT
This incident affected: Management (Management Portal), Voice Services (Call Completion), Client Applications (Desktop), Fax Services (Fax Completion), and Integrations.