All Nodes - Enterprise Monitoring Point loss of Experience and Delivery monitoring data

AppNeta Engineering has applied software fixes in last night's deployment to address the issues referenced in this status page. At this point, Monitoring Points are connecting back , have the proper statuses applied in APM, and Monitoring Point upgrades are re-enabled. We consider this issue resolved and will post the results of our postmortem.

AppNeta Engineering has taken steps to address the sequencers stuck in an upgrading state and has confirmed that the telemetry is flowing for majority of these these as well. We will leave Monitoring Point upgrades disabled through the weekend and as we produce long-term fixes for the issues that have lead to this.

We will continue to monitor this issue going forward.

Maintenance against App-03 and App-13 application nodes is now complete.

AppNeta Engineering is going to perform emergency maintenance against the App-03 and App-13 application nodes, in relation to this issue. During this brief time (1-2 minutes), APM will be unavailable. This will take place at 6:15pm (EST)

In an effort to avoid more instances of failed Monitoring Point upgrades, AppNeta Engineering is disabling the ability to upgrade Monitoring Points on demand.

While Monitoring Point connections and telemetry have remained stable through the night, we have identified an issue with some Monitoring Points, which has caused them to stall in their upgrade. During this time, those monitoring points will be inactive, not generating telemetry or running diagnostics. AppNeta Engineering is working on this issue and will post updates here.

Emergency maintenance is complete and we can see monitoring data coming in on application nodes. We will continue to monitor this issue.

AppNeta Engineering will be performing emergency maintenance across all application nodes, to address this issue. APM will be briefly interrupted for a short period of time across the next 30 minutes.

AppNeta is experiencing a loss of Experience and Delivery monitoring data from Enterprise Monitoring Points. AppNeta Engineering is investigating the issue.

We will provide updates as we work through the issue.