HCP Cluster Operations Delayed
Resolved·Full outage

We have confirmed that HCP services have reach healthy system state and are functioning as intended

Sun, Jan 29, 2023, 10:26 PM
(2 years ago)
·
Affected components
HCP Portal
Updates

Resolved

We have confirmed that HCP services have reach healthy system state and are functioning as intended

Sun, Jan 29, 2023, 10:26 PM

Monitoring

HCP engineers have implemented a remediation to alleviate high latency with create/update/delete action for the platform and we are observing latency levels return to normal. We will continue to monitor the platform to make sure latency stays within acceptable bounds.

Sun, Jan 29, 2023, 09:24 PM(1 hour earlier)

Monitoring

HCP engineers have identified the cause of high latency with create/update/delete actions for the platform and are currently working on a remediation.

Sat, Jan 28, 2023, 07:38 PM(1 day earlier)

Monitoring

We are seeing successful create/update/delete actions for the platform but high latencies are still present. HCP engineers will continue to monitor the health of the platform.

Sat, Jan 28, 2023, 05:32 AM(14 hours earlier)

Identified

We are seeing successful create/update/delete actions for the platform but high latencies are still present. HCP engineers will continue to monitor the health of the platform.

Sat, Jan 28, 2023, 02:24 AM(3 hours earlier)

Identified

The updates have been applied to our production systems and we are waiting to verify the success of these changes

Sat, Jan 28, 2023, 01:25 AM(59 minutes earlier)

Identified

We have begun to make the aforementioned changes in production, and thus HCP create/update/destroy actions will be unavailable.This outage will last 30 minutes

Sat, Jan 28, 2023, 12:30 AM(54 minutes earlier)

Identified

Through our testing in lower environments we have determined that our fix will resolve the degradation of HCP's functionality. This change will result in a 30 minute outage, to be experience in the failures of create/update/delete actions.

The change is to occur at 00:30UTC/16:30PST

Fri, Jan 27, 2023, 11:55 PM(35 minutes earlier)

Investigating

HCP Engineers are continuing to troubleshoot the issue have discovered the path forward to resolution. To ensure the minimal impact to our service offerings we are testing the resolution in lower environments. An update will be posted when changes are to be made in production.

Customers may experience elevated latency on actions (create/update/delete) for their HCP resources prior to the fix being applied. When the changes are made to production there will be an outage where any create/update/delete action will fail. This outage is expected to last 30 minutes

Fri, Jan 27, 2023, 11:12 PM(42 minutes earlier)

Investigating

We are currently aware of operations such as create, updates and deletes of clusters within HCP that may take longer to complete, investigation is ongoing. Customer Consul and Vault clusters should remain operational.

Thu, Jan 26, 2023, 11:33 PM(23 hours earlier)

Investigating

We are currently aware of operations such as create, updates and deletes of clusters within HCP that may take longer to complete. Customer Consul and Vault clusters should remain operational.

Thu, Jan 26, 2023, 10:45 AM(12 hours earlier)