Primevox Communications - Localized System Outage – Incident details

All systems operational

Localized System Outage

Resolved
Partial outage 5 %
Started 1 day agoLasted about 1 hour

Affected

Voice Services

Partial outage from 6:29 PM to 7:27 PM

Outbound Calling

Partial outage from 6:29 PM to 7:27 PM

AnswerCARE & Voicemail

Partial outage from 6:29 PM to 7:27 PM

Inbound Calling

Partial outage from 6:29 PM to 7:27 PM

Updates
  • Postmortem
    Postmortem

    Permanent Corrective Actions

    We have already begun deploying upgraded routing infrastructure across our datacenters to further improve resiliency and failover performance. The updated architecture includes:

    • Dedicated IP address space owned and managed directly by our organization
    • Multi-carrier BGP routing to provide additional carrier redundancy
    • Enhanced inter-datacenter traffic failover capabilities

    These improvements significantly reduce reliance on any single upstream network carrier and strengthen platform resiliency during carrier congestion or outage scenarios.

    Our purple platform already operates on this architecture, which is why it remained fully operational throughout this incident.

    Preventive Improvements

    In addition to the infrastructure upgrades already underway, we are implementing several operational improvements designed to strengthen automated failover validation and incident detection, including:

    • Enhanced verification of PBX evacuation completion status
    • Improved detection of partially completed failover events
    • Additional safeguards for system failover during severe packet loss conditions
    • Expanded monitoring and alerting for evacuation inconsistencies and failover anomalies

  • Resolved
    Resolved

    Our development team manually identified the remaining affected PBX systems and migrated them to healthy datacenters, restoring service availability for impacted customers.

    The upstream network carrier later repaired the affected infrastructure responsible for the packet loss condition within the Phoenix datacenter, fully stabilizing network connectivity.

  • Identified
    Identified

    Major packet loss within our Phoenix datacenter caused service disruption for a portion of customers on our green platform. Our purple platform was not impacted.

    As designed, our systems automatically began moving affected customer phone systems to another datacenter to minimize impact. While the migration process initially reported as successful, we later identified that a limited number of systems did not fully transition due to instability within the affected server environment causing a disruption to service for those customers.