Forecall: lessons from a linter for MCP tool descriptions

Where it started

Building and using a few MCP servers, I kept watching agents pick the wrong tool or get the argument shape wrong. The model sees only what tools/list returns: a name, a description and an input schema. How a tool is described is how it gets called. And yet there was no linter for descriptions.

Scoring public servers

I started by scoring eight well-known MCP servers, 126 tools in all, with one ruler. Average scores per server ranged from 33 to 70. Notion’s official server alone had 57 pairs of tools whose descriptions were nearly the same.

The textbook case is the filesystem server’s list_directory and list_directory_with_sizes, with a description similarity of 0.86. Both say what they do; neither says when to use which. From the agent’s side it is a coin toss. I reported the two, with proposed rewrites, as modelcontextprotocol/servers#5071.

How the linter scores

Each tool gets up to 100 points across six parts.

Part	Points	What it looks for
Purpose	20	Not a restatement of the name, not too short
When to use	20	When to use it, when not to, what to use instead
Arguments	25	A description per argument, constraints or examples for strings, required
Return value	15	What comes back
Constraints and side effects	10	Auth, rate limits, side effects, annotations
Examples	10	A concrete call

Server-wide issues are reported apart from per-tool ones: too many tools, near-identical descriptions, shared descriptions. Without that split, “a similar description exists” lands on every tool and becomes noise.

The scoring rules are versioned (v2 today). Any change that can give some input a different score or finding raises the version and updates the fixtures. People gate CI on these numbers, and a score that moves silently is the worst outcome.

npx forecall dump -o tools.json -- node build/index.js   # take tools/list from your own server
npx forecall lint tools.json                              # score it; nothing is sent anywhere

@forecall/lint has no dependencies and never touches the network. The GitHub Action forecall/forecall-cli@v1 scores every pull request into the job summary and fails the step on min-tool-score or fail-on.

Beyond the linter

A static score and whether a real model actually picks the tool are two different things. So two more pieces exist.

Evaluations. For each tool, requests such as “find the largest file” are written, then Claude, GPT and Gemini choose and call for real, and Forecall measures how often the right tool is called with the right arguments. Public servers’ tool lists sit alongside as distractors.

Failure KB. An MCP server where agents look up known failures of MCP tool calls and their workarounds before and after a call. A failure one agent reports becomes verified only after agents of two other teams reproduce it. The KB is a remote Streamable HTTP server; for clients that can only start servers over stdio there is a small relay, forecall-mcp.

Whether the static score and the measurements actually correlate is something a calibration tool keeps checking, using the same case generation and calls as the evaluations. A high score does not guarantee being chosen.

How it is built

Everything runs on Cloudflare Workers. The API is Hono, the public site and the dashboard are React Router v8, evaluations run as Workflows with one instance per suite, and rate limits live in a Durable Object. The database is Neon Postgres (pgvector) through Hyperdrive, and the vendors’ models are called through AI Gateway. Only the linter and CLI are split out into the public repository forecall/forecall-cli; the service pins the npm releases.

What I learned

• Near-identical descriptions are a server problem, not a tool problem. Report at the wrong level and nobody fixes it
• The rules version is part of the output. Say so when it changes
• The first scoring rules were a Python research script. The scores of the eight servers were frozen as fixtures before the port to TypeScript, which became the source of truth from v2. The fixtures are what prove the port right

Paste a tools/list JSON at forecall.dev to get a shareable result page, no sign-up needed.