BLF Outage
Between Thursday, October 22nd at 5:13 AM CST and Friday, October 23rd at 6:34 PM CST, Cytracom BLF functionality was impacted while undergoing routine maintenance on a core component interacting with the BLF service. The result of this procedural failure was continual "soft" failures within the BLF service. The normal recovery reactions resulted in an overload of traffic and consumption of capacity.
Corrective measures have been implemented in several areas including but not limited to: maintenance procedures, rate-limiting for BLF nodes and capacity planning for the addition of future nodes.