trackmcp
All posts
ProductJun 9, 2025·5 min read

How to A/B test your MCP tool descriptions

Tool descriptions are UX for agents. Here's how to test wording changes and prove which version gets tools used correctly.

Krishna GoyalKrishna GoyalFounder, TrackMCP
Key takeaways
  • Descriptions are UX for agents — test them, don't guess.
  • Change one description at a time to attribute the effect.
  • Measure adoption, correctness, and retries.

Because agents choose tools by reading descriptions, wording is one of your highest-leverage levers. And like any UX copy, it should be tested rather than guessed.

What to test

  • The verb and object in the tool name
  • When-to-use guidance in the description
  • Argument names and the example shapes you show

How to run it

Change one description at a time so you can attribute the effect. Note the change and the date, then let real traffic run against it for a meaningful window.

What to measure

  • Adoption: does call volume for the tool rise?
  • Correctness: does success rate hold or improve?
  • Retries: do give-up loops go down?

Keep what wins

If adoption rises and success holds, the new wording was better. Treat descriptions as a living surface you iterate on, and underused tools often come back to life.

See this on your own server

TrackMCP turns your MCP server's calls into adoption, workflows, and outcomes. One line to install.

Keep reading