Improve retrieve_tools diagnostics for annotation filters and automation-critical tools #969

Summary

I am using MCPProxy as a local lightweight gateway for many MCP servers, with retrieve_tools as the main way to keep tool schemas out of the agent context until needed. This works very well overall, but I ran into a failure mode where a tool existed and was callable, yet was effectively hidden from routine automation because annotation-based filtering removed it without a very actionable explanation.

I would like to propose a small set of discovery/diagnostic improvements rather than a large architecture change.

Real-world failure mode

A finance automation needed read-only tools from an upstream server, e.g. a tool equivalent to:

wx-copilot:finance_get_capabilities

The upstream server was healthy, tool refresh worked, and direct qualified calls worked. However, retrieve_tools(..., read_only_only=true) did not return the expected finance tools because the upstream tools were missing MCP annotations.readOnlyHint=true.

After the upstream MCP server was fixed to publish annotations like:

{
  "readOnlyHint": true,
  "destructiveHint": false,
  "openWorldHint": true
}

retrieve_tools(..., read_only_only=true) started returning the tools correctly again.

So the root cause was upstream metadata, but from the operator/agent side the experience looked like tool discovery had failed.

Suggested improvements

1. Diagnostics for tools filtered by annotation gates

When a query uses filters such as read_only_only=true, exclude_destructive=true, or exclude_open_world=true, it would be very helpful if the response included a compact diagnostic count, for example:

{
  "filter_diagnostics": {
    "matched_before_filters": 12,
    "omitted_missing_readOnlyHint": 8,
    "omitted_destructiveHint": 2,
    "omitted_openWorldHint": 0,
    "suggestion": "Some tools matched the query but were omitted because readOnlyHint was missing or false. Check upstream annotations or retry without read_only_only for diagnosis."
  }
}

This would make it clear that the tool exists but is being filtered for safety, instead of leaving the user to infer that the server/tool is unavailable.

2. Better upstream-scoped retrieval

A way to scope discovery to a known upstream would reduce ambiguity in large installations:

{
  "query": "finance capabilities",
  "server": "wx-copilot",
  "read_only_only": true
}

or equivalent behavior through query syntax, profile, or server weighting.

This is useful when there are hundreds or thousands of tools and the operator already knows which upstream should satisfy the request.

3. Pinned or required tool IDs for automation preflight

For recurring automations, it would be useful to define a small list of expected tool IDs and have MCPProxy validate their availability and risk metadata explicitly, e.g.:

{
  "required_tools": [
    "wx-copilot:finance_get_capabilities",
    "wx-copilot:finance_search_review_items",
    "wx-copilot:finance_search_obligations",
    "wx-copilot:finance_search_movements"
  ],
  "require_read_only": true
}

This could be a new tool, a mode of retrieve_tools, or a profile-level health check. The goal is not to bypass security. The goal is to fail with a precise explanation when an automation-critical tool is missing, quarantined, unindexed, or lacks required annotations.

Why this fits MCPProxy

I am deliberately not proposing a heavier enterprise gateway model. MCPProxy is valuable to me because it is local, lightweight, uses a single binary/web UI, and focuses on token reduction plus safe discovery. These suggestions seem aligned with that design:

• Better explanation when safety filters remove tools.
• Better targeting when the desired upstream is known.
• Better preflight for automations that depend on a small stable set of tools.

Possible implementation shape

A small first PR could focus only on item 1:

• Track candidate tools before and after annotation filters.
• Return compact diagnostic counts when filters remove otherwise relevant matches.
• Include one short suggestion string only when diagnostic counts are non-zero.

Then upstream-scoped retrieval and required-tool preflight could be discussed separately.

Related context

This is especially important for read-only financial, accounting, operational, or production automations where the operator wants strict filters enabled, but also needs to know whether a missing tool is:

• Genuinely unavailable.
• Quarantined.
• Not indexed yet.
• Ranked too low.
• Filtered because upstream annotations are missing or incorrect.

Thanks for building MCPProxy. The retrieval-first model is a strong fit for local multi-MCP workflows; this issue is mostly about making the safe failure modes more legible.