Study Finds Limits to Coding Agents Learning from Revised Instructions

Researchers Christoph Treude of Singapore Management University and Sebastian Baltes of Heidelberg University tested whether coding agents improve when they revise their instructions after completing tasks. In seven tests where the target problem recurred, two instruction changes eliminated it. Some changes also added checks to unrelated tasks and increased median runtime. The authors said project-wide guidance may remain useful, while notes about specific task failures may not transfer to later work.

Generated by AI from reports · Updated 15 minutes ago

There is not enough continuous observation data to plot a trend yet.