Duration: 3 hours and 34 minutes

Affected Components: Inbound Calling, Outbound Calling, AnswerCARE & Voicemail

Aug 27, 17:00:00 GMT+0
Investigating - We are currently investigating intermittent packet loss affecting network traffic between some of our data centers. This may result in intermittent connectivity issues or degraded call quality for some customers. Services remain operational, and we will provide another update as soon as we have more information..

Aug 27, 17:53:04 GMT+0
Identified - We have identified the cause of the intermittent packet loss as an issue with the fiber carrier serving our Dallas data center. Our team is working with the carrier to resolve the issue and restore normal network connectivity between our data centers. Services remain operational, but some customers may continue to experience intermittent connectivity or degraded call quality while the issue is being addressed..

Aug 27, 20:19:04 GMT+0
Monitoring - Our fiber carrier has implemented a fix for the issue affecting connectivity between our data centers. Our team is monitoring the network closely to confirm the fix has fully resolved the intermittent packet loss..

Aug 27, 20:34:00 GMT+0
Resolved - This incident has been resolved..

Duration: 1 hour

Affected Components: Inbound Calling

Aug 14, 13:15:00 GMT+0
Investigating - We are currently investigating reports of customers not receiving inbound calls. Our team has confirmed the issue and is actively working to identify the root cause. We'll provide an update as soon as we know more..

Aug 14, 13:44:00 GMT+0
Identified - We've identified the issue: one of our partner carriers is experiencing a partial outage causing SIP calls not to be delivered to our network as expected. We are working directly with the carrier to resolve this and have begun rerouting traffic to minimize impact while a permanent fix is put in place..

Aug 14, 13:55:35 GMT+0
Monitoring - A fix has been implemented in coordination with our partner carrier, and inbound calling has been restored. We are closely monitoring call traffic to confirm the issue is fully resolved and stable..

Aug 14, 14:15:00 GMT+0
Resolved - This incident has been resolved. Inbound calling is functioning normally, and we've confirmed stability with our partner carrier. We apologize for any disruption this caused and appreciate your patience..

Duration: 58 minutes

Affected Components: Inbound Calling, Outbound Calling, AnswerCARE & Voicemail

May 13, 18:29:00 GMT+0
Identified - Major packet loss within our Phoenix datacenter caused service disruption for a portion of customers on our green platform. Our purple platform was not impacted. As designed, our systems automatically began moving affected customer phone systems to another datacenter to minimize impact. While the migration process initially reported as successful, we later identified that a limited number of systems did not fully transition due to instability within the affected server environment causing a disruption to service for those customers..

May 13, 19:27:00 GMT+0
Resolved - Our development team manually identified the remaining affected PBX systems and migrated them to healthy datacenters, restoring service availability for impacted customers. The upstream network carrier later repaired the affected infrastructure responsible for the packet loss condition within the Phoenix datacenter, fully stabilizing network connectivity..

May 13, 23:28:24 GMT+0
Postmortem - ### **Permanent Corrective Actions** We have already begun deploying upgraded routing infrastructure across our datacenters to further improve resiliency and failover performance. The updated architecture includes: • Dedicated IP address space owned and managed directly by our organization • Multi-carrier BGP routing to provide additional carrier redundancy • Enhanced inter-datacenter traffic failover capabilities These improvements significantly reduce reliance on any single upstream network carrier and strengthen platform resiliency during carrier congestion or outage scenarios. Our purple platform already operates on this architecture, which is why it remained fully operational throughout this incident. ### **Preventive Improvements** In addition to the infrastructure upgrades already underway, we are implementing several operational improvements designed to strengthen automated failover validation and incident detection, including: • Enhanced verification of PBX evacuation completion status • Improved detection of partially completed failover events • Additional safeguards for system failover during severe packet loss conditions • Expanded monitoring and alerting for evacuation inconsistencies and failover anomalies.

Duration: 1 day, 15 hours and 21 minutes

Affected Components: AnswerCARE & Voicemail

Apr 2, 09:10:21 GMT+0
Resolved - This incident has been resolved..

Mar 31, 17:49:34 GMT+0
Investigating - We’re currently investigating reports of voicemail transcriptions not being generated as expected and voicemail emails not being sent. Our team is actively working to identify the root cause and restore normal functionality..

Mar 31, 19:11:58 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Mar 31, 19:11:17 GMT+0
Identified - We are continuing to work on a fix for this incident..

Duration: 2 hours and 54 minutes

Affected Components: AnswerCARE & Voicemail

Feb 19, 16:28:15 GMT+0
Identified - We are currently experiencing a voicemail service disruption due to packet loss between data centers. Our team is actively working to restore voicemail services as quickly as possible. We will provide updates as progress is made..

Feb 19, 16:31:27 GMT+0
Monitoring - The voicemail service disruption caused by packet loss between data centers has been resolved. Services are fully restored at this time. We are actively monitoring the system to ensure continued stability. Please contact support if you experience any further issues..

Feb 19, 19:22:00 GMT+0
Resolved - This incident has been resolved..

Duration: 1 hour and 35 minutes

Affected Components: PBX Admin & User Portal

Dec 8, 20:30:09 GMT+0
Identified - We have identified an issue with our web portal. Our engineering team is actively working to resolve this as quickly as possible. .

Dec 8, 21:00:27 GMT+0
Monitoring - Access to our web portal has been fully restored. Our engineering team has implemented a fix and services are operating as expected. We are continuing to monitor the portal to ensure stability. An internal investigation is underway to determine the underlying cause and to prevent future occurrences. We will provide additional updates if necessary. Thank you for your patience. If you continue to experience any issues, please contact our support team..

Dec 8, 22:05:39 GMT+0
Resolved - This incident has been resolved..

Duration: 1 hour and 2 minutes

Affected Components: Call History API, AnswerCARE & Voicemail

Aug 14, 20:46:50 GMT+0
Identified - We are currently experiencing a partial outage in our NYC data center (one of our five data centers). This is impacting Call History reporting and web portal access for customers served from this location. Our engineering team is actively working with the data center to restore services as quickly as possible. All other call services remain fully operational, and customers in other regions are not affected. We will provide updates as soon as more information becomes available..

Aug 14, 20:52:11 GMT+0
Monitoring - The earlier issue affecting Call History reporting and web portal access for customers in our NYC data center has been resolved. All services are now operating normally, and we continue to monitor for stability..

Aug 14, 21:48:23 GMT+0
Resolved - This incident has been resolved..

Duration: 7 minutes

Affected Components: eFax API, SMS/MMS Messaging API, Inbound Calling, Outbound Calling, Click-to-Call API, Call History API, PBX Admin & User Portal, , Web Portal & APIs →

Jul 29, 19:21:50 GMT+0
Identified - Hello, we are currently observing some significant packet loss in our Raleigh, NC datacenter. We've engaged our fiber carriers to determine the cause and fix it as soon as possible. In the meantime, call quality for some customers may be affected, and accessing the web portal/fax portal may experience delays..

Jul 29, 19:29:11 GMT+0
Resolved - Re-routing around the outage was successful, we are seeing everything return to normal. If you are still encountering any issues, please send an email to support@primevox.net.

Duration: 17 hours and 15 minutes

Affected Components: Outbound Messaging

Apr 10, 13:36:32 GMT+0
Resolved - This incident has been resolved, all outbound SMS/MMS messaging is functional..

Apr 9, 20:21:53 GMT+0
Identified - SMS providers and carriers nationwide have identified an issue affecting the delivery of outbound SMS messaging traffic. At this time, inbound SMS messaging traffic remains unaffected. We will provide updates on this page as soon as the issue with outbound SMS traffic is resolved..

Apr 10, 03:10:32 GMT+0
Monitoring - The stream carrier responsible for the SMS/MMS delivery issues has identified the root cause and implemented a fix. At this time, SMS/MMS messages appear to be sending successfully. We will continue to actively monitor the situation to ensure continued stability..

Duration: 4 hours

Affected Components: Inbound Calling, Outbound Calling

Nov 17, 04:00:01 GMT+0
Identified - Maintenance is now in progress.

Nov 17, 08:00:00 GMT+0
Completed - Maintenance has completed successfully.

Nov 17, 04:00:00 GMT+0
Identified - Our development team will be adding additional SIP servers to our infrastructure to accommodate customer growth. This is planned as a no-downtime operation, so service should remain uninterrupted. **Important:** If you or your customers use a managed firewall that requires whitelisting for VoIP traffic, please contact [support@primevox.net](mailto:support@primevox.net) to request a list of the new IP addresses associated with these upgrades..

Duration: 1 hour and 27 minutes

Affected Components: Outbound Messaging

Nov 13, 16:57:53 GMT+0
Identified - Our team has been notified of an issue with an upstream carrier impacting outbound messaging. The carrier is actively working on a resolution, and we will provide updates as soon as they become available..

Nov 13, 17:25:08 GMT+0
Monitoring - Our team has been notified that the issue with the upstream carrier impacting outbound messaging has been resolved. We are currently monitoring messaging traffic for any additional problems..

Nov 13, 18:25:08 GMT+0
Resolved - This incident has been resolved..

Duration: 1 hour and 20 minutes

Affected Components: eFax API, Outbound Messaging, Inbound Messaging, Click-to-Call API, SMS/MMS Messaging API, Call History API, Inbound Calling, Outbound Calling, PBX Admin & User Portal, AnswerCARE & Voicemail

Oct 23, 13:44:38 GMT+0
Identified - We are observing some major packet loss across multiple internet carriers, which may be impacting voice quality and web portal access. All teams are engaged and working on the issue to get it resolved ASAP..

Oct 23, 14:06:45 GMT+0
Identified - Public internet disruptions have been resolved, so voice quality and portal access is back to normal. Viewing Call History (CDR Events) via the web portal and API is still impacted, and will be resolved as quickly as possible. For now, call history will not be viewable temporarily..

Oct 23, 15:05:06 GMT+0
Resolved - Call history viewing and APIs are now functional. At this time, all services are back to normal..

Duration: 4 hours

Affected Components: eFax API, Outbound Messaging, Inbound Messaging, Click-to-Call API, SMS/MMS Messaging API, Call History API, PBX Admin & User Portal

Oct 19, 13:30:01 GMT+0
Identified - Maintenance is now in progress.

Oct 19, 17:30:00 GMT+0
Completed - Maintenance has completed successfully.

Oct 19, 13:30:00 GMT+0
Identified - This Saturday, October 19 at 8:30am Central Time to 12:30pm Central Time, our Dallas datacenter will be upgraded. During this time window, the Web Portal and APIs **may go offline** for some of the maintenance window. This includes features like Digital Faxing, Messaging, and phone system PBX management/reporting. This maintenance window **will not** affect phones, apps, or voice traffic - phone calls and phone systems will continue to operate normally..

Duration: 2 hours

Affected Components: Inbound Calling, Outbound Calling

Sep 25, 06:00:00 GMT+0
Completed - Maintenance has completed successfully.

Sep 25, 04:00:01 GMT+0
Identified - Maintenance is now in progress.

Sep 25, 04:00:00 GMT+0
Identified - We will be performing scheduled maintenance to increase server capacity and enhance system performance. We expect little to no downtime; however, there may be brief interruptions as phones fail over between data centers. We appreciate your patience and understanding as we work to improve our network and your experience..

Duration: 49 minutes

Affected Components: AnswerCARE & Voicemail

Sep 10, 16:28:44 GMT+0
Identified - We have identified an issue where voicemail emails are not being sent. This issue does not affect AnswerCARE. An immediate solution is to Apply Changes on the affected PBX. Otherwise, a system-wide update will be implemented tonight after hours..

Sep 10, 17:17:20 GMT+0
Resolved - This incident has been resolved. All voicemail-to-email notifications should be functioning properly..

Duration: 2 hours and 10 minutes

Affected Components: eFax API, Outbound Messaging, Inbound Messaging, Click-to-Call API, SMS/MMS Messaging API, Call History API, PBX Admin & User Portal

May 26, 13:10:01 GMT+0
Identified - We have identified a power issue in our Dallas datacenter, and an on-site technician is currently working on the issue. This is affecting the web portal and APIs, which includes digital faxing and SMS/MMS messaging..

May 26, 15:20:04 GMT+0
Resolved - Power to Dallas has been restored, and web portal access is functional. .

Duration: 17 hours and 5 minutes

Affected Components: AnswerCARE & Voicemail

Apr 8, 14:55:54 GMT+0
Identified - We've identified an issue with custom voicemail greetings playback and our development team is currently working on a fix..

Apr 8, 18:53:20 GMT+0
Identified - Our development team has created a solution for the custom voicemail playback issue. However, to avoid any service interruptions, its implementation has been scheduled for 11:00 PM ET tonight..

Apr 9, 08:00:49 GMT+0
Resolved - Voicemail greeting playback has been fixed, and voicemail functionality is fully restored. We apologize for this inconvenience, and testing measures have been created to ensure a similar voicemail failure cannot occur in the future..

Duration: 3 hours and 31 minutes

Affected Components: Inbound Calling, Outbound Calling

Feb 2, 16:16:51 GMT+0
Identified - We have identified some internet congestion in our NYC datacenter, which can cause quality issues for a handful of customers. We are currently working with our fiber optic teams to restore normal connectivity. Customers located in this datacenter have already been moved to an alternative datacenter to help mitigate this issue. The Realtime Dashboard (and Realtime Queue Analytics) may show inconsistent or absent behavior. This is expected..

Feb 2, 19:48:08 GMT+0
Resolved - This incident has been resolved..

Feb 2, 17:19:08 GMT+0
Monitoring - The fiber optic outage (it was a national outage by backbone provider Cogent, affecting many websites and services) is largely repaired and appears to be resolved. We will continue to monitor closely. There may still be occasional call quality issues, but these should be very few and far between. Thank you for your patience..

Duration: 21 hours and 51 minutes

Affected Components: Inbound Calling, Outbound Calling

Jan 18, 20:55:02 GMT+0
Identified - We are investigating some call quality issues in our New York datacenter, and will be moving all affected customers out of this datacenter shortly. Affected customers may experience a temporary call drop as the failover takes place, but overall downtime will not exceed 10 seconds. If any phone BLFs (light-up buttons) have issues synchronizing, simply reboot the affected phone..

Jan 18, 21:17:41 GMT+0
Monitoring - Affected customers have been moved out of the problematic datacenter, and we are seeing call quality return to normal. We will continue to monitor actively for any changes. If any BLFs (light-up buttons) on a phone are not working properly, simply reboot the phone to sync it back up with our servers. Thank you for your patience and understanding!.

Jan 19, 18:46:17 GMT+0
Resolved - We were notified by our NYC datacenter that the impaired fiber line has been fixed around 4:00am Eastern Time this morning. Since then, we have observed full stability and quality..

Duration: 15 days, 19 hours and 18 minutes

Affected Components: Outbound Calling

Feb 2, 16:23:42 GMT+0
Resolved - This incident has been resolved. Updating the app from Google Play will fix it..

Jan 17, 21:06:02 GMT+0
Identified - After the recent release of Cloud Softphone's version 6.2.3 update on the Google Play Store, Android users have reported that they are unable to make outbound calls. Our development team identified that this version is causing communication issues with encrypted servers, such as Primevox's. Fortunately, Apple and desktop users (Mac/Windows) are unaffected. We've reported the issue to Acrobits, the developer of Cloud Softphone, and updates are on the way. For now, we kindly ask Android users to avoid updating the app and recommend disabling automatic app updates until a fix is implemented..

Duration: 9 minutes

Affected Components: Inbound Calling, Outbound Calling, AnswerCARE & Voicemail

Nov 3, 17:54:25 GMT+0
Resolved - This incident has been resolved. We have routed around the affected national fiber carrier..

Nov 3, 17:45:49 GMT+0
Investigating - There is a significant amount of internet congestion occurring between the East and West coasts, which may cause some customers to experience call quality issues. We have reported this to our upstream fiber providers and a solution is being implemented..

Duration: 6 hours and 1 minute

Affected Components: Inbound Calling, Outbound Calling

Oct 25, 13:37:03 GMT+0
Investigating - There is a significant amount of internet congestion occurring between the East and West coasts, which may cause some customers to experience call quality issues. We have reported this to our upstream fiber providers and a solution is being implemented..

Oct 25, 14:09:33 GMT+0
Identified - We have identified the internet provider responsible for the congestion and are actively implementing measures to reroute our voice traffic around this provider..

Oct 25, 14:48:57 GMT+0
Monitoring - A fix has been implemented and we are currently monitoring the situation..

Oct 25, 19:38:21 GMT+0
Resolved - This incident has been resolved..

Duration: 6 hours and 3 minutes

Affected Components: Inbound Calling, Outbound Calling

Sep 6, 21:07:25 GMT+0
Investigating - There is a significant amount of internet congestion occurring between the East and West coasts, which may cause some customers to experience call quality issues. We have reported this to our upstream fiber providers and a solution is being implemented..

Sep 7, 03:10:14 GMT+0
Resolved - This incident has been resolved..

Sep 6, 21:29:20 GMT+0
Monitoring - Internet congestion has been mitigated and everything appears to be stabilized. We will continue monitoring closely should any issue arise..

Duration: 22 hours and 51 minutes

Affected Components: Inbound Calling, Outbound Calling

Aug 29, 15:21:11 GMT+0
Resolved - This incident has been resolved..

Aug 28, 16:30:00 GMT+0
Identified - Due to some internet congestion between the West and East coasts, there may be occasional issues with call quality for a handful of customers. The fiber providers involved in the congestion have been notified and are working on a fix now..

Duration: 15 minutes

Affected Components: Outbound Calling

Aug 7, 18:00:00 GMT+0
Identified - This incident is being opened retroactively. Outbound calls may not be completing for some users..

Aug 7, 18:15:00 GMT+0
Resolved - This incident has been resolved. A major upstream carrier experienced an incident causing some user's calls not to be connected. Traffic was re-routed to an alternative carrier allowing calls to being completed normally again..

Duration: 38 minutes

Affected Components: Inbound Calling, Outbound Calling

Jul 28, 16:47:34 GMT+0
Resolved - This incident has been resolved..

Jul 28, 16:09:23 GMT+0
Identified - Due to some internet congestion along the East Coast, there may be intermittent voice quality issues for some customers. All of our upstream fiber carriers have been notified of the issue, and are working on a resolution..

Duration: 9 hours and 39 minutes

Affected Components: PBX Admin & User Portal

Apr 3, 17:33:17 GMT+0
Identified - An issue has been identified with our **Real-Time Latency** and **Extension Status Snapshot** reports. These reports are currently displaying online extensions as "offline". This problem is isolated to reporting only, and the functionality of extensions showing as "offline" is not affected: they are functioning normally. We have a pending fix that will be applied to these reports this evening..

Apr 4, 03:12:43 GMT+0
Resolved - Real-Time Latency and Extension Snapshot reports have been fixed, and should now be displaying correct online/offline extension status. Thank you for your patience..

Duration: 1 hour

Affected Components: PBX Admin & User Portal

Mar 3, 05:00:00 GMT+0
Completed - Maintenance has completed successfully.

Mar 3, 04:00:00 GMT+0
Identified - Tonight at 10:00pm Central Time, we will be publishing some upgrades to the following systems: - Real-Time Dashboards - Webhooks During the upgrade, real-time dashboards such as Realtime Queues and Realtime Latency may experience delays or show inaccurate status. After the upgrade, customers should not notice any changes with Webhook integrations, however if any issue is suspected with webhooks, please send an email to support@primevox.net and we will look into it. If you do not utilize our APIs or Webhooks, you may ignore this notice..

Mar 3, 04:00:01 GMT+0
Identified - Maintenance is now in progress.

Duration: 1 hour and 50 minutes

Affected Components: PBX Admin & User Portal, SMS/MMS Messaging API, Call History API, Click-to-Call API, eFax API

Oct 25, 22:46:01 GMT+0
Identified - Our Dallas datacenter has experienced a loss of fiber link, impacting our web portal. Phone systems and phones have failed over to alternative datacenters, however our eFax and web portal are unusable. Our Dallas fiber carriers have been notified and we are working to get the web portal online as quickly as possible..

Oct 26, 00:36:28 GMT+0
Resolved - Our datacenter provider has confirmed connectivity in Dallas, so our Web Portal is back online. If you encounter any issues with phones, a simple reboot (unplug the power cable, and plug it back in) can help..

Duration: 12 minutes

Dec 12, 17:46:30 GMT+0
Identified - Maintenance is now in progress..

Dec 12, 17:56:33 GMT+0
Completed - Remediation of the log4j vulnerability has been completed..

Dec 12, 17:45:00 GMT+0
Identified - We have identified a single piece of software in our infrastructure that is susceptible to the newly discovered log4j 0-day vulnerability, and will be patching it shortly. There is no expected impact to services..