Runs Failing to Start with Error: Cannot provision execution environment.

Summary

On July 6, 2026, runs failed to start for approximately 22 minutes, returning the error "Cannot provision execution environment." The root cause was a stuck PVC (persistent volume claim) during a migration to new network storage for the Terraform/OpenTofu providers cache.

Impact

Between 16:48 and 16:53 UTC, new runs across affected workspaces failed to schedule with a "Cannot provision execution environment" error.

Timeline (all times UTC)

• 16:48 — Issue detected; investigation began
• 16:50 — Root cause identified: PVC stuck due to in-use protection finalizer
• 16:51 — Mitigation deployed (providers cache disabled); monitoring began
• 16:53 — Continued monitoring, no further impact observed
• 17:10 — Incident resolved

What Happened

During the migration to new network storage for the Terraform/OpenTofu providers cache, the existing PVC was still attached to multiple active workloads. Kubernetes' in-use protection finalizer blocked deletion and recreation of the volume, so new runs could not bind to it and failed to schedule.

Mitigation

We disabled the providers cache to unblock run scheduling while we worked on the underlying volume issue. This restored run scheduling immediately, with only a minor, temporary performance impact from running without a warm provider cache.

Root Cause

The migration runbook did not account for the PVC's in-use protection finalizer when workloads were still actively bound to the volume, which blocked the planned delete-and-recreate step.

Remediation and Prevention

• Immediate: delete the workloads still holding the PVC, recreate the volume, and restore it as the providers cache.
• Preventive: update the migration runbook to explicitly drain or cordon any workloads still bound to a PVC before attempting to delete and recreate it, rather than assuming the volume is free.