From Plan to Action: How Well Do Agents Follow the Plan?

Agents aspire to eliminate the need for task-specific prompt crafting through autonomous reason-act-observe loops. Still, they are commonly instructed to follow a task-specificplanfor guidance, e.g., to resolve software issues following phases for navigation, reproduc- tion, patch, and validation. Unfortunately, it is unknown to what extent agents actually follow such instructed plans. Without such an analysis—determining the extent agentscomplywith a given plan—it is impossible to assess whether a solution was reached through correct strategic reasoning or through other means, e.g., data contamination or overfitting to a benchmark. This paper presents the first extensive, systematic analysis of plan compliancein programming agents, examining 16,991 trajecto- ries from SWE-agent across four LLMs on SWE-bench Verified and SWE-bench Pro under eight plan variations. Without an explicit plan, agents fall back on workflows internalized during training, which are often incomplete, overfit, or inconsistently applied. Pro- viding the standard plan improves issue resolution, and we observe that periodic plan reminders can mitigate plan violations and im- prove task success. A subpar plan hurts performance even more than no plan at all. Surprisingly, augmenting a plan with additional task-relevant phases in the early stage can degrade performance, particularly when these phases do not align with the model’s internal problem-solving strategy. These findings highlight a research gap: fine-tuning paradigms that teach the models to follow in- structed plans, rather than encoding task-specific plans in them. This requires teaching the model to reason and act adaptively, rather than memorizing workflows.