Feb 2, 16:16:51 GMT+0
Identified -
We have identified some internet congestion in our NYC datacenter, which can cause quality issues for a handful of customers. We are currently working with our fiber optic teams to restore normal connectivity.
Customers located in this datacenter have already been moved to an alternative datacenter to help mitigate this issue.
The Realtime Dashboard (and Realtime Queue Analytics) may show inconsistent or absent behavior. This is expected..
Feb 2, 19:48:08 GMT+0
Resolved -
This incident has been resolved..
Feb 2, 17:19:08 GMT+0
Monitoring -
The fiber optic outage (it was a national outage by backbone provider Cogent, affecting many websites and services) is largely repaired and appears to be resolved.
We will continue to monitor closely.
There may still be occasional call quality issues, but these should be very few and far between.
Thank you for your patience..
Jan 18, 20:55:02 GMT+0
Identified -
We are investigating some call quality issues in our New York datacenter, and will be moving all affected customers out of this datacenter shortly.
Affected customers may experience a temporary call drop as the failover takes place, but overall downtime will not exceed 10 seconds.
If any phone BLFs (light-up buttons) have issues synchronizing, simply reboot the affected phone..
Jan 18, 21:17:41 GMT+0
Monitoring -
Affected customers have been moved out of the problematic datacenter, and we are seeing call quality return to normal. We will continue to monitor actively for any changes.
If any BLFs (light-up buttons) on a phone are not working properly, simply reboot the phone to sync it back up with our servers.
Thank you for your patience and understanding!.
Jan 19, 18:46:17 GMT+0
Resolved -
We were notified by our NYC datacenter that the impaired fiber line has been fixed around 4:00am Eastern Time this morning. Since then, we have observed full stability and quality..
Feb 2, 16:23:42 GMT+0
Resolved -
This incident has been resolved. Updating the app from Google Play will fix it..
Jan 17, 21:06:02 GMT+0
Identified -
After the recent release of Cloud Softphone's version 6.2.3 update on the Google Play Store, Android users have reported that they are unable to make outbound calls. Our development team identified that this version is causing communication issues with encrypted servers, such as Primevox's. Fortunately, Apple and desktop users (Mac/Windows) are unaffected.
We've reported the issue to Acrobits, the developer of Cloud Softphone, and updates are on the way. For now, we kindly ask Android users to avoid updating the app and recommend disabling automatic app updates until a fix is implemented..
Nov 3, 17:45:49 GMT+0
Investigating -
There is a significant amount of internet congestion occurring between the East and West coasts, which may cause some customers to experience call quality issues.
We have reported this to our upstream fiber providers and a solution is being implemented..
Nov 3, 17:54:25 GMT+0
Resolved -
This incident has been resolved. We have routed around the affected national fiber carrier..
Oct 25, 13:37:03 GMT+0
Investigating -
There is a significant amount of internet congestion occurring between the East and West coasts, which may cause some customers to experience call quality issues.
We have reported this to our upstream fiber providers and a solution is being implemented..
Oct 25, 14:09:33 GMT+0
Identified -
We have identified the internet provider responsible for the congestion and are actively implementing measures to reroute our voice traffic around this provider..
Oct 25, 14:48:57 GMT+0
Monitoring -
A fix has been implemented and we are currently monitoring the situation..
Oct 25, 19:38:21 GMT+0
Resolved -
This incident has been resolved..
Sep 6, 21:29:20 GMT+0
Monitoring -
Internet congestion has been mitigated and everything appears to be stabilized.
We will continue monitoring closely should any issue arise..
Sep 7, 03:10:14 GMT+0
Resolved -
This incident has been resolved..
Sep 6, 21:07:25 GMT+0
Investigating -
There is a significant amount of internet congestion occurring between the East and West coasts, which may cause some customers to experience call quality issues.
We have reported this to our upstream fiber providers and a solution is being implemented..
Aug 29, 15:21:11 GMT+0
Resolved -
This incident has been resolved..
Aug 28, 16:30:00 GMT+0
Identified -
Due to some internet congestion between the West and East coasts, there may be occasional issues with call quality for a handful of customers.
The fiber providers involved in the congestion have been notified and are working on a fix now..
Aug 7, 18:00:00 GMT+0
Identified -
This incident is being opened retroactively. Outbound calls may not be completing for some users..
Aug 7, 18:15:00 GMT+0
Resolved -
This incident has been resolved.
A major upstream carrier experienced an incident causing some user's calls not to be connected. Traffic was re-routed to an alternative carrier allowing calls to being completed normally again..
Jul 28, 16:47:34 GMT+0
Resolved -
This incident has been resolved..
Jul 28, 16:09:23 GMT+0
Identified -
Due to some internet congestion along the East Coast, there may be intermittent voice quality issues for some customers.
All of our upstream fiber carriers have been notified of the issue, and are working on a resolution..
Apr 3, 17:33:17 GMT+0
Identified -
An issue has been identified with our **Real-Time Latency** and **Extension Status Snapshot** reports. These reports are currently displaying online extensions as "offline". This problem is isolated to reporting only, and the functionality of extensions showing as "offline" is not affected: they are functioning normally.
We have a pending fix that will be applied to these reports this evening..
Apr 4, 03:12:43 GMT+0
Resolved -
Real-Time Latency and Extension Snapshot reports have been fixed, and should now be displaying correct online/offline extension status.
Thank you for your patience..
Mar 3, 04:00:00 GMT+0
Identified -
Tonight at 10:00pm Central Time, we will be publishing some upgrades to the following systems:
- Real-Time Dashboards
- Webhooks
During the upgrade, real-time dashboards such as Realtime Queues and Realtime Latency may experience delays or show inaccurate status.
After the upgrade, customers should not notice any changes with Webhook integrations, however if any issue is suspected with webhooks, please send an email to support@primevox.net and we will look into it.
If you do not utilize our APIs or Webhooks, you may ignore this notice..
Mar 3, 04:00:01 GMT+0
Identified -
Maintenance is now in progress.
Mar 3, 05:00:00 GMT+0
Completed -
Maintenance has completed successfully.
Oct 25, 22:46:01 GMT+0
Identified -
Our Dallas datacenter has experienced a loss of fiber link, impacting our web portal.
Phone systems and phones have failed over to alternative datacenters, however our eFax and web portal are unusable.
Our Dallas fiber carriers have been notified and we are working to get the web portal online as quickly as possible..
Oct 26, 00:36:28 GMT+0
Resolved -
Our datacenter provider has confirmed connectivity in Dallas, so our Web Portal is back online.
If you encounter any issues with phones, a simple reboot (unplug the power cable, and plug it back in) can help..
Aug 4, 19:39:13 GMT+0
Identified -
There have been some packet loss incidents reported along fiber routes between Phoenix and Miami, resulting in occasional call quality issues or dropped calls.
Our upstream fiber carriers have been notified and are working to route around the ISP issues now..
Aug 4, 21:06:29 GMT+0
Resolved -
The affected fiber providers identified the bad route and repaired it, so connection quality between MIA and PHX is back to normal.
If any phones are still misbehaving, a simple reboot will fix them: unplug the power cord (so the phone completely shuts off), wait 10 seconds, then plug it back in.
Thank you for your patience!.
Jun 7, 21:36:14 GMT+0
Identified -
Hello, we have identified some internet backbone issues around our Miami, FL data center. This means a small subset of phones may experience degraded call quality.
We have notified our fiber carriers in that region, and they are looking into the issue now..
Jun 7, 22:39:42 GMT+0
Resolved -
At this time, we are seeing normal connectivity in our Miami datacenter. The cause of the packet loss was a fiber cut with one of our upstream providers (Cogent) and they have restored full connectivity.
We will continue to monitor closely, and thank you for your patience!.
Feb 11, 15:24:43 GMT+0
Identified -
Hello, we've identified packet loss in our Phoenix, AZ datacenter.
Some customers my experience dropped calls and an inability to access our web portal.
All engineers are actively engaged to repair this issue as quickly as possible..
Feb 11, 15:31:03 GMT+0
Resolved -
The packet loss in our Phoenix datacenter has been resolved. Calls, phone status, and web portal activity should all return to normal within minutes.
We genuinely apologize for this inconvenience!.
Jan 14, 15:07:00 GMT+0
Identified -
Several major upstream carriers are experiencing delays with SMS/MMS (text and picture messaging). These messages may be delayed in delivery or experience delivery failures. Not all carriers are affected.
If you are unable to receive MFA/2FA while trying to log in to the Cloud PBX portal, please send an email to support@primevox.net to have us temporarily disable this feature..
Jan 14, 19:54:21 GMT+0
Identified -
Inbound SMS and MMS messaging is operating normally.
Outbound SMS and MMS, however, continue to experience some delays with delivery. Upstream carriers are still working on a resolution..
Jan 14, 20:54:52 GMT+0
Resolved -
Upstream carriers are showing fully operational SMS and MMS messaging.
If you continue to encounter any issues with messaging, please email support@primevox.net and we will be happy to assist!.
Dec 16, 16:28:59 GMT+0
Resolved -
The packet loss in our Raleigh, NC datacenter has been fixed, and we are seeing that traffic return to normal.
What this means: Very few users likely noticed any issue at all. Even though this was a small event, PrimeVOX feels it is important to always maintain clear and proactive communications with our customers, just in case a small event would ever lead to a larger one, we want you to always know what is going on..
Dec 16, 15:55:08 GMT+0
Identified -
We have identified some packet loss coming out of our Raleigh, NC datacenter. Our fiber carriers are engaged and working to resolve this issue.
What this means: Packet loss can sometimes cause occasional calls to drop, or call quality to become degraded for some calls. Most people will not notice this issue since it is limited to only 1 data center..
Dec 12, 17:45:00 GMT+0
Identified -
We have identified a single piece of software in our infrastructure that is susceptible to the newly discovered log4j 0-day vulnerability, and will be patching it shortly.
There is no expected impact to services..
Dec 12, 17:46:30 GMT+0
Identified -
Maintenance is now in progress..
Dec 12, 17:56:33 GMT+0
Completed -
Remediation of the log4j vulnerability has been completed..