After monitoring the fix implemented on 29 March 2019, we have confirmed that Terraform runs are executing as expected and all error rates have stabilized.

We have implemented a fix for this issue that will be gradually rolled out over the next 24 hours. We will continue to monitor Terraform runs and their service ecosystem through the weekend, in order to ensure Terraform Enterprise stabilizes after the full roll out of the fix. We will update this incident at or before 9AM PDT on 1 April 2019 to confirm the efficacy of the fix.

Error rates have stabilized and returned to normal range in the services responsible for executing Terraform runs. We are continuing to monitor these services to ensure that Terraform run functionality is fully restored. We will provide an update once we are confident all Terraform runs will plan and apply as expected.

We are currently investigating a spike of errors affecting a portion of jobs running in Terraform Enterprise. As a result some users may see failed or stalled Terraforrm Runs. We will provide updates as our investigation develops.