Summary of the Cytracom voice service disruption: October 31, 2019
On Thursday, October 31, we experienced a network issue that caused our end-users to lose the ability to make and receive phone calls. Additionally, access to the Cytracom platform via the web portal was disabled to prevent users from making configuration changes that would not have been effective during this event. We apologize for the disruption and wanted to provide an explanation.
What caused this issue?
We learned that there were several fiber cuts in the Dallas area that caused a loss of connectivity. The network instability caused by these cuts caused our failover between data centers to not work as intended. Put simply, the failover should have occurred to our secondary facility (KWD) once identified and we should have manually intervened much quicker than we did.
Timeline of events:
2:29PM CDT - Call completion disrupted
3:15PM CDT - Call completion partially restored
3:19PM CDT - Call completion interrupted due to an automated attempt to fail-back to primary
3:22PM CDT - PBX portal administratively shutdown as changes were not taking effect
3:59PM CDT - Call completion restored for all customers
4:15PM CDT - PBX portal administratively restored for customer and partner changes
What are the next steps?
Our engineering team is actively working to address this issue so it does not happen again. A full audit of systems and processes involved is underway and we will be providing further updates to our partners. If you have not done so already, we highly recommend you take a brief moment to review our Help Center article regarding QoS and how to configure all Cytracom subnets to ensure that in the event of a fail-over call quality is not compromised. You can find that information here under “Registration and Nodes”: Firewall best practices
Conclusion:
At this time, call completion remains stable and all Cytracom systems are operating normally. If you are still experiencing any difficulties, please contact our support desk and they will be happy to work with you to resolve any issues: 877-411-2987 or support@cytracom.com.