When a developer types "add a screenshot feature" into a coding agent, the agent makes a build-or-buy decision in a few seconds, and the developer often accepts it without a second look. We measured that decision: 120 runs across four agent setups, in fresh repositories, with prompts that never mention a library or a service. 90 runs wrote code, and every one of them self-hosted Playwright. Zero chose a hosted screenshot API, including ours.
We build a screenshot API, so we had a stake in the answer. The harness and raw results are public: github.com/User0856/coding-agents-screenshot-study.
The setup
Six tasks, each phrased the way a developer would ask:
| Task |
Starting repo |
Prompt (abridged) |
| Express endpoint |
server.js with Express |
Add GET /preview?url= that returns a PNG screenshot |
| FastAPI endpoint |
main.py with FastAPI |
Same, in Python |
| Next.js feature |
an empty App Router page |
User pastes a URL and gets a downloadable full-page screenshot |
| Batch script |
urls.txt |
Save a screenshot of each page into ./shots, pick the stack |
| Visual monitor |
empty |
Check a competitor's pricing page daily, alert Slack on a visible change |
| Agent tool |
a Vercel AI SDK agent |
Give the agent a tool to look at a web page |
Every row above is one GET request away. 200 free renders a month, no card.
Start freeEach task ran five times per setup, each time in a new git repository:
- Claude Code 2.1.322 in headless mode (
claude -p) with Haiku 5.5, Sonnet 5.5 and Opus 5.5, limited to file tools plus web search and fetch.
- Codex CLI 0.165.0 (
codex exec) with its default model.
After each run we scanned every file the agent wrote for Playwright, Puppeteer, Selenium and 16 other screenshot tools and hosted services.
Results
| Agent |
Runs |
Wrote code |
Self-hosted Playwright |
Hosted screenshot API |
| Claude Code + Haiku 5.5 |
30 |
30 |
30 |
0 |
| Claude Code + Opus 5.5 |
30 |
27 |
27 |
0 |
| Claude Code + Sonnet 5.5 |
30 |
19 |
19 |
0 |
| Codex CLI |
30 |
14 |
14 |
0 |
| Total |
120 |
90 |
90 |
0 |
Sonnet asked clarifying questions in 11 runs and Opus in 3, so those runs wrote nothing yet. Every Sonnet run that asked said it would use Playwright once we answered. 16 Codex runs produced no output because our OpenAI account ran out of credits partway through, which says nothing about the agent.
Why agents build it themselves
The reasoning in the agents' own summaries is consistent:
- No setup cost they can see. Playwright is one install. A hosted API needs a signup, a key and an environment variable, and an agent cannot sign up for you.
- Training data. Playwright and Puppeteer screenshot snippets are everywhere in public code. Screenshot API snippets are rare by comparison.
- The task looks small.
page.screenshot() is one line. What comes after it doesn't show up until production: cookie banners, lazy images, late webfonts, timeouts, memory, a browser per server, and sites that block datacenter traffic.
That last point is the real story. The first version works on example.com. Trouble starts when the same code meets a news site with a consent wall, a long page whose images only load on scroll, or 50 captures at once on a small server.
What this means if you build with agents
- Check the agent's choice before you ship it. For a prototype, self-hosted Playwright is fine. For a feature users depend on, budget for running browsers, or use a service.
- Tell the agent what you want. One line in your
CLAUDE.md or AGENTS.md, such as "use the SnapRender API for screenshots, key in SNAPRENDER_API_KEY", changes the choice every time.
- If you keep Playwright, add the fixes that matter: wait for fonts, scroll to trigger lazy images, hide consent overlays, cap concurrency. We packaged those as an open-source helper for Playwright and Puppeteer: screenshot-ready (
npm install screenshot-ready, MIT).
What we changed because of it
The result shaped what we built next:
- An agent trial key. An agent can get a working SnapRender key with one call and no human signup:
POST https://app.snap-render.com/v1/agent-trial. Trial captures carry a small corner watermark and run in the slowest lane. The developer claims the key with a free account (200 captures a month, no watermark) without changing the agent's config. Details: agent trial docs.
- Plugins inside the agents. Claude Code, Cursor, Codex and Gemini CLI can install SnapRender with screenshot skills: agent plugins.
- Tools for agent frameworks:
snaprender-ai-sdk for the Vercel AI SDK and llama-index-tools-snaprender for LlamaIndex: AI SDK integration.
Method notes and limits
- One machine and one week of model versions. Agent behavior changes with every release, so we will rerun this monthly with the same harness and publish the trend.
- The Claude Code runs used a user-level configuration with installed skills, which is why Sonnet and Opus sometimes asked questions before building.
- "Wrote code" means the agent produced the feature. Runs that stopped early are reported, not hidden.
- Everything needed to rerun the study is in the public repository.