All Systems Operational
Atlas ? Operational
Packer Builds ? Operational
Terraform Runs ? Operational
Logging ? Operational
SCADA ? Operational
File Storage ? Operational
GitHub Operational
AWS ec2-us-east-1 Operational
AWS ec2-us-west-1 Operational
Twilio Outgoing SMS Operational
Twilio REST API Operational
Application compilation ? Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
System Metrics Month Week Day
Atlas Frontend Public Availability ?
Fetching
File Storage Availability ?
Fetching
Logging Availability ?
Fetching
SCADA Availability
Fetching
Past Incidents
Apr 24, 2017

No incidents reported today.

Apr 23, 2017

No incidents reported.

Apr 22, 2017

No incidents reported.

Apr 21, 2017

No incidents reported.

Apr 20, 2017
Resolved - Additional capacity has been brought online and the queue is back to normal.
Apr 20, 22:01 UTC
Update - Still working on adding capacity; queues are making progress but still backed up.
Apr 20, 21:26 UTC
Monitoring - Due to high demand, we're adding additional backend capacity to assist in restoring proper queue times.
Apr 20, 20:59 UTC
Investigating - Terraform runs are queueing past acceptable time ranges and we're investigating the cause.
Apr 20, 20:50 UTC
Apr 19, 2017

No incidents reported.

Apr 18, 2017

No incidents reported.

Apr 17, 2017
Resolved - Due to underlying AWS failure, we had a Vault leader transition resulting in 15 seconds of errors in encrypting and decrypting variables. These errors were returned from 2017-04-17 18:27:45 - 2017-04-17 18:28:00 UTC. We are following up with tasks to add retry behavior to downstream clients so we can sustain Vault leader transitions and recover without returning errors to users.
Apr 17, 18:53 UTC
Investigating - We're investigating a spat of application-level errors stemming from an internal Vault service interruption affecting variable decrypt/encrypt operations. The batch of errors appears to be limited to a short timespan, but we will follow up as we learn more.
Apr 17, 18:38 UTC
Apr 16, 2017

No incidents reported.

Apr 15, 2017

No incidents reported.

Apr 14, 2017

No incidents reported.

Apr 13, 2017
Resolved - All services are operating normally.
Apr 13, 17:11 UTC
Monitoring - We have repaired the issue with Packer Build log retrieval for builds during the problematic window. All Packer Build log retrieval should now be working. We will continue to monitor closely for errors.
Apr 13, 15:40 UTC
Update - We have completed an application rollback that addresses problems downloading Vagrant boxes over 1GB. We are investigating some issues with Packer Build log retrieval for builds executing during the rollback window.
Apr 13, 15:04 UTC
Identified - We've identified the root cause and are working on a resolution.
Apr 13, 14:19 UTC
Investigating - We are currently investigating this issue.
Apr 13, 13:52 UTC
Apr 12, 2017

No incidents reported.

Apr 11, 2017

No incidents reported.

Apr 10, 2017

No incidents reported.