Someone hands you an MCP server. A vendor's remote endpoint, or an open-source one they found on a directory, and the ask is whether the team can point it at the company workspace. You have an afternoon.
Most of page one for this query is written for the person who wrote the server: a test pyramid, unit tests around your own tool handlers, fixtures, a CI story. Good advice, and none of it available to you, because you do not own the code and are not going to read all of it. What you need is an evaluation: a short ordered sequence that ends in a verdict.
How do you test an MCP server you did not write?
You evaluate its surface rather than its code. Enumerate every tool, sort them into reads and writes, connect against a workspace you can throw away, call one tool correctly and one deliberately wrong, and check what trace the calls left behind. Six checks, in that order, ending in a yes or a no.
The procedure, in order
Each row produces a verdict about one thing. Run them in sequence, because a failure early makes everything below it untestable.
| # | What you check | How you check it | A pass looks like |
|---|---|---|---|
| 1 | The tool surface | Connect in a scratch client profile and list the tools. Count them, then read every name and description | Each name is distinguishable from every other one, and each description says when to call it, not just what it does |
| 2 | Reads versus writes | Sort the list into read, write, and destructive by name and input schema. Write the three counts down | The read set alone covers the job you actually wanted. Nothing is ambiguous enough that you had to guess which pile it belongs in |
| 3 | Blast radius | Point it at a seeded demo account or a throwaway workspace, never at production, for the whole evaluation | The workspace is one you would delete without asking anyone's permission |
| 4 | A successful call | Ask one real question in your own words, the way a colleague would phrase it | Typed records with ids, units and currency, not a paragraph of prose the model has to re-parse |
| 5 | A failed call | Call with a malformed id, an empty filter, and one request you should not be entitled to | Three distinct, legible errors. No stack trace, no other tenant's rows, no silent empty success |
| 6 | The trail | After the calls, go looking for the record of them: a server-side log, an audit view, or an identity tool | Someone other than the person who made the calls can reconstruct who called what |
1. Enumerate before you connect anything real
The first artifact is a list, and you write it down. Connect the server in a scratch client profile, ask for the tool list, and read it end to end. This is the one step where a dedicated inspector beats a chat client, because it shows you the raw surface without a model standing between you and it.
MCP Inspector is the usual choice here, and it is the official one. npx @modelcontextprotocol/inspector connects to the server the way a client would and gives you every tool, resource and prompt it exposes, a form for calling each one by hand, and the raw JSON-RPC traffic underneath. Nothing is choosing tools on your behalf, which is exactly the property you want while you are still deciding whether to trust the thing.
Two things fall out immediately. The count tells you what you are adding to every future conversation, and the names tell you whether a model will pick correctly under pressure. Names like search and query sitting next to each other are a retrieval failure waiting to happen. Descriptions that explain the endpoint rather than the situation ("returns account objects") will get called at the wrong moment.
Count the tools yourself rather than trusting a number in a README or a vendor benchmark. The list comes from the protocol; the README comes from marketing.
2. Sort reads from writes, on paper
Take the list and split it into three piles: reads, writes, and anything that deletes or is irreversible. Do this by hand, from the names and the input schemas, before you connect to anything you care about.
The pile sizes are the finding. If the read set alone does the job you were asked to enable, you have just discovered that the write half is optional, and optional access is access you decline in week one. If a tool is ambiguous enough that you cannot place it, that ambiguity is the model's problem too, and it is a mark against the surface.
This is also the point where a server's honesty is cheapest to check. A server that claims a read-only mode should have a read-only tool list to show for it, not a promise in the docs.
3. Run it against something you can throw away
Everything from here on is live calling, so it happens somewhere disposable. A vendor's own seeded demo workspace is ideal, because it is populated enough for the answers to be meaningful and belongs to nobody. Failing that, a scratch project you created this morning.
The rule is blunt: for the whole evaluation, the data on the other end is data you would delete without asking. Whether the server eventually deserves production data is the trust decision, and that has its own procedure. This step exists so the evaluation itself is free.
Worked example, since ours is built to be checked this way. Run npx -y @aioproductoscom/mcp@latest with no PRODUCTOS_TOKEN set, and the stdio server starts in demo mode against a fully seeded showcase workspace with a read-only subset of the tools. No account, no credentials, no signup. What you should get back is a tool list from the protocol itself and typed records from the first call, and what you should not get back is a successful write, because those tools are not on a tokenless surface.
4. Ask one real question, then read the shape of the answer
Now make one call, phrased the way a teammate would phrase it rather than the way the schema is written. What you are grading is not whether it answered. It is what came back.
Structured records with ids let you make a second call. Units and currency let a model do arithmetic without inventing a denomination. An explicit empty result ("no accounts matched, 0 of 412 scanned") stops the model from filling the silence with a plausible explanation. Prose does none of that, and a server that answers in prose will quietly degrade every conversation it is attached to.
Watch the size too. A tool that dumps several thousand rows into the context window has spent the budget the model needed for reasoning. If the description does not mention pagination, assume there is none.
5. Break it on purpose
This is the step nobody runs, and it is the one that separates a careful server from a careless one. Make three calls you expect to fail: a malformed or non-existent id, a filter that matches nothing, and something you should not be entitled to.
You are looking for three different errors, each of which tells you something. A not-found should say not found, not return an empty list that reads as a legitimate zero. A permission denial should say denied rather than returning someone else's rows. And none of the three should hand back a stack trace, a raw database error, or a fragment of another tenant's data, because a model will happily read all of that back to whoever is in the conversation.
A server that returns the same vague failure for all three has told you it does not distinguish between them internally either. If the calls do not go through at all, that is a connection problem rather than a quality one, and the five-layer diagnosis is a different job from this one.
6. Ask what it left behind
Last check, and the one that gets asked after an incident rather than before one: go looking for the record of the calls you just made.
For a hosted server, that means a server-side log or an audit view, and the question is whether an administrator who was not in the conversation can reconstruct who called what. If the server exposes an identity or whoami tool, call it, because "which account was that" is often the whole answer. For a local server running as a subprocess on one laptop, the honest finding is usually that there is no trail that outlives the process, which is a real property of that architecture rather than a bug, and it is covered in what a local server costs at team scale.
Write the answer down either way. "Nowhere" is a legitimate result; it just means this server does not get production data.
When this procedure is overkill
Three cases where running all six checks is the wrong use of an afternoon.
The server is local-only and touches nothing shared. A filesystem server, a browser driver, a device on your desk. There is no tenancy to isolate and no audit trail to demand, because the blast radius is one laptop that is already logged into everything. Read the source or pin the version, and skip the rest.
You wrote it. Then page one is right and this post is not for you. Unit-test the handlers, add fixtures, wire it into CI. The reason the procedure above exists is precisely that none of those options are on the table for code you did not write.
You are comparing five servers rather than approving one. Full evaluation is too slow at that width. Run steps 1 and 2 on all of them, which is cheap, and only take the shortlist through the live calls. Surveying what is out there first is a different task, and the roundup is the faster start.
The procedure is deliberately mechanical because the alternative is a vibe. Enumerate, sort, sandbox, call, break, trace, decide. If you want to run it against a production surface where every tool is documented before you connect, ours is listed tool by tool, on flat tiers with no per-seat metering.