Design a coding agent
An engineer files an issue at 6pm: "Checkout shows the old price after a coupon is removed." By the time they are back at their desk the next morning, there is a pull request waiting. It has a failing test that reproduces the bug, a three line fix, the test now passing, and a description explaining the cause. They review it, ask for one rename, and merge.
Behind that is an agent loop, a fleet of sandboxes, and a lot of careful engineering about what a model is allowed to touch. This question is showing up in interviews at every company building developer tools, and the strong answers are about the sandbox and the verification, not the model.
Step 1: Understand the problem#
| You ask | They say | What it settles |
|---|---|---|
| Interactive in the editor, or in the background from an issue? | Background. It takes an issue and opens a pull request for a person to review. | Long running async jobs instead of a chat, a sandbox per task, and the pull request as the only output. Human code review becomes the final safety net, which makes some risks acceptable that would not be in an autonomous system. |
| How big is the codebase? | A monorepo of about two million lines across several languages. | It cannot fit in any context window, so the agent needs tools to search and navigate code, and a strategy for keeping its context small over a long task. |
| What can it run? | Builds, tests and linters. Never production systems or production credentials. | An isolated sandbox with the repo checked out and dependencies installed, network access limited to package registries, and no secrets that matter. |
| How many tasks, and how long does each take? | About 3,000 tasks a day across 2,000 engineers, each running for 10 to 40 minutes. | Concurrency: how many sandboxes run at once at the Monday morning peak, which sets the compute budget and the model rate limit. |
| How do we know it worked? | The change builds, tests pass, and reviewers merge it without major rework. | Tests as the agent's own feedback loop, a historical eval set of real issues with their real fixes, and merge rate as the production metric. |
| Who can write issues? | Employees, but issues often paste in logs, stack traces and text from customer tickets. | Issues are untrusted input. A pasted log can contain instructions, so the sandbox must be safe even if the agent is fully fooled. |
What you are building, and what you cut#
- Take an issue and produce a pull request. With a description of the cause and the fix.
- Run an agent loop with code tools. Search, read, edit, run commands, all inside a sandbox.
- Verify its own work. Reproduce the bug with a test, fix it, run the suite.
- Let engineers steer or stop a task. Comments on the task reach the running agent.
- A sandbox ready within 30 seconds of a task starting.
- No path from a sandbox to production data or credentials.
- Hard budget per task: steps, tokens, wall clock time.
- Every action recorded, so a reviewer can see how the change was made.
- Merging. A person merges, always, in this design.
- Deploying. Out of the sandbox, out of scope.
- Training a code model. Assume a strong hosted model.
Back of the envelope#
Each model call resends the growing context, so input tokens dominate and prompt caching is not optional: without it this bill is several times larger. The number that decides whether the product is worth it is cost per merged pull request, not cost per task.