Following the 7.0.10 release to the NA region several of our Web Socket Rendezvous Points were not accepting new connections despite surface level metrics indicating the service was healthy, and actively serving other clients. As customers came online for the day we saw a spike in 400/500 errors on specific nodes and began steps to troubleshoot and remediate. Several of the issues identified in this incident were patched with the 7.0.20 hotfix and additional monitoring has been put in place in an effort to detect this failure sooner post-release.
Posted Jan 30, 2025 - 23:35 CST
Resolved
This incident has been resolved.
Posted Jan 09, 2025 - 10:29 CST
Update
A fix has been implemented and we are monitoring the results.
Posted Jan 09, 2025 - 10:00 CST
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Jan 09, 2025 - 09:48 CST
Identified
The issue has been identified and a fix is being implemented.
Posted Jan 09, 2025 - 09:42 CST
Investigating
We are investigating reports of remote tools issues in app.ninjarmm.com