How Large Language Models Generate Infrastructure Code
Roxane Fischer, CEO of AnyShift.io and former AI researcher, explains the fundamental mechanics of how LLMs work and their application to infrastructure as code. Neural networks learn patterns through training on massive datasets, encoding information into mathematical representations that enable them to predict the next most likely token in a sequence. When applied to Terraform code generation, these models use probabilistic prediction to suggest configurations based on patterns learned from public repositories. However, the presentation reveals a critical limitation: infrastructure code is severely underrepresented in training data, with only 2 million HCL files on GitHub compared to over 32 million Python files—a 32x difference. This data scarcity means models often miss best practices, generate hard-coded values instead of proper resource dependencies, and lack the live context of production infrastructure that would enable them to generate enterprise-grade configurations.
Security Risks and the Probabilistic Nature Problem
The presentation highlights serious security concerns with AI-generated infrastructure code. Because LLMs are probabilistic rather than deterministic, they can propagate vulnerabilities found in their training data, such as overly permissive security group rules with open ports (0.0.0.0/0). More concerning is the potential for adversarial attacks: if malicious actors publish modules with security flaws or malicious providers to GitHub, subsequent model retraining could incorporate these patterns, leading AI assistants to recommend compromised configurations. Fischer emphasizes that the probabilistic nature of neural networks means they never generate code with 100% certainty, making deterministic security scanning tools like Checkov and Snyk essential safeguards. The risk extends beyond simple misconfigurations to potential credential theft through malicious provider imports that models might suggest based on poisoned training data.
Synthesis AI vs. Generative AI: Different Tools for Different Jobs
Fischer draws a crucial distinction between two AI paradigms in DevOps. Generative AI takes minimal input and creates new content—an open-ended process with a large solution space that's prone to hallucination and inaccuracy. Synthesis AI, by contrast, takes large amounts of existing data and extracts insights from it, offering higher accuracy because the solution is contained within the input. For infrastructure operations, synthesis AI excels at log analysis and root cause analysis, finding patterns across millions of log entries or correlating customer alerts with system logs across heterogeneous data sources. This approach is already proving valuable in tools like Google Cloud Ops AI, which can identify the needle in the haystack by recognizing patterns that human operators might miss. Understanding when to use each approach is critical for effective AI adoption in infrastructure management.
The Future: Context-Aware AI Through Graph RAG
The presentation concludes with Fischer's vision for overcoming current limitations through context-aware AI systems. The solution lies in Retrieval Augmented Generation (RAG) technology, specifically Graph RAG, which treats infrastructure as an interconnected graph of resources rather than flat documents. Traditional RAG encodes company knowledge into searchable vector representations, but infrastructure requires understanding relationships between VPCs, subnets, IAM roles, and other resources. Graph RAG constructs a knowledge graph where nodes represent resources and edges represent relationships, enabling AI to query based on actual infrastructure topology rather than simple text similarity. The core challenges are constructing meaningful relationship definitions (how a VPC connects to subnets differs from tag-based connections) and efficiently traversing this graph at query time. When combined with LLMs, this context-aware approach could finally enable AI to generate infrastructure code with proper dependencies, security configurations, and enterprise-grade practices tailored to specific environments.
