The replacement load balancer is performing as expected. We will use any information that stems from the root cause analysis with AWS to improve our practices going forward.
Posted Nov 21, 2017 - 20:32 UTC
Monitoring
We will continue to work with AWS to pursue root cause analysis of the problematic load balancer, but in the meantime we have provisioned a replacement load balancer which is now in service. We are monitoring closely, but we expect this will resolve the issue at hand.
Posted Nov 21, 2017 - 19:17 UTC
Identified
The root cause of the errors seems to be a single AWS Load Balancer listener getting into a state where it times out all connections. We believed that removing and re-adding this listener was enough to get the out of this failure mode, but the timeouts eventually returned. We have now removed the listener again and we are working with AWS Support to determine best next steps.
Posted Nov 21, 2017 - 18:23 UTC
Monitoring
We believe we have identified and addressed the root cause of the application errors. We are monitoring the situation and will update if things change.
Posted Nov 21, 2017 - 17:30 UTC
Investigating
We are still encountering application errors and we are working on establishing the root cause.
Posted Nov 21, 2017 - 17:09 UTC
Monitoring
Between 15:15 UTC and 15:50 UTC, Terraform Enterprise had a high rate of application errors. We have taken steps to remediate the errors and the error levels have returned to normal. We are investigating the root cause.