Cloud infrastructure drift silently creates security vulnerabilities and cost explosions when engineers bypass Terraform for hotfixes

technology0 views
When engineers make emergency changes directly through the AWS Console, Azure Portal, or GCP Console instead of through Terraform or other Infrastructure as Code tools, the actual cloud infrastructure silently diverges from the declared IaC configuration. These 'hotfixes' — a security group rule opened during an incident, an instance type upgraded during a traffic spike — are rarely backported to IaC, creating a growing gap between what the code says the infrastructure is and what it actually is. Why it matters: the declared infrastructure state becomes a lie, so terraform plan produces unreliable diffs that engineers stop trusting, so teams stop running terraform apply entirely out of fear of breaking production, so manual console changes become the norm rather than the exception, so the organization loses all the auditability, reproducibility, and security guarantees that justified adopting IaC in the first place. The structural root cause is that the IaC workflow is optimized for planned changes, not emergency response — running a Terraform change through code review and CI/CD takes 30-60 minutes while clicking in the console takes 30 seconds, so under incident pressure, engineers rationally choose speed over process every time.

Evidence

Manual changes through cloud provider consoles are identified as the primary cause of infrastructure drift in every major IaC survey. HashiCorp's own documentation acknowledges that drift is inevitable at scale and recommends running terraform plan -refresh-only to detect it, but most organizations do not run drift detection on a regular schedule. Spacelift, Terramate, env0, and ControlMonkey have all built drift detection products specifically because the problem is so pervasive. The longer drift goes unaddressed, the more difficult realignment becomes — organizations report that after 6+ months of undetected drift, reconciliation can take weeks of engineering time. Sources: HashiCorp Terraform drift detection documentation, Spacelift drift detection guide (2025), Terramate drift remediation guide (2025), env0 drift detection guide.

Comments