One minor correction, VCS connection issues can be mitigated by downgrading tfc-agent from 1.33.2 to 1.33.1.

VCS connection issues should be mitigated by downgrading to tfc-agent 1.33.2. Our engineering team is still investigating the cause of this failure. If you do not have any VCS connection problems you do not need to downgrade to 1.33.1. We will release a new version of tfc-agent in the next 24 hours to fix the root cause.

Downgrading tfc-agent to 1.33.1 has proven effective in eliminating the intermittent VCS failures. We will continue to monitor for any further impacts. A new release with a fix is in-progress and expected within 24 hours. If you are not affected, do not downgrade. If you experience further issues please contact support.

Our engineering team has identified the tfc-agent release v1.33.2 as the cause of intermittent issues with VCS providers. We have downgraded the version to 1.33.1 for remote runs. If you are experiencing issues with self-hosted agents please downgrade to 1.33.1 until a new release is made to address the issue.

Current impact state: Remote runs fully-functional. Affected self-hosted agents need to downgrade to tfc-agent version 1.33.1. Stacks users may experience a regression when using proxies with version 1.33.1.

Action underway: Releasing a fixed tfc-agent version.

Next update time: 19:45 UTC

Intermittent and some persistent VCS issues have been reproduced on some workspaces. Our engineering team is continuing to investigate.

Current impact state: Intermittent issues on VCS workspaces are continuing to be observed.

Action underway: Engineering team is working on reliably reproducing the issue.

Next update time: 19:00 UTC

Our engineers are no longer able to reproduce the issue and error counts have dropped off. We will continue to monitor for any further issues. If you experience any further problems please reach out to support.

Current impact state: System in recovery.

Action underway: Monitoring.

Next update time: 19:00 UTC

Our engineering is continuing to investigate intermittent VCS-related failures.

Current impact state: VCS related workloads experiencing intermittent issues.

Action underway: Engineering team investigating.

Next update time: 18:30 UTC

We are continuing to receive reports of intermittent issues with VCS workloads, some of which are not hosted on GitHub.com. Our engineering team is investigating these issues.

Current impact state: VCS workloads experiencing intermittent issues.

Action underway: Awaiting word from GitHub.com

Next update time: 18:00 UTC