MCP Quality Evidence vs. Star Ratings: What Should Agents Trust?
Why AI-agent service selection needs task-specific evidence, version identity, independent sources, recency, and uncertainty rather than a universal star score.
Short answer
Star ratings are useful discovery signals, but agents should not treat them as execution evidence. For selection, use observations tied to the exact endpoint, version, task, constraints, time window, and independent sources—and expose uncertainty when the evidence is weak.
The failure this prevents
Humans can interpret a four-star average with context and skepticism. An agent may convert it into a deterministic choice even when reviews concern different versions, use cases, regions, or incentives.
Recommended workflow
- Use stars and directories to create a candidate list, not to make the final decision.
- Resolve each candidate to an exact endpoint and version.
- Compare evidence for the actual task and constraints.
- Inspect independent-source count, recency, corrections, and missing dimensions.
- Fall back to testing, an approved list, or human review when evidence is insufficient.
Working starting point
Evidence checklist: - exact canonical endpoint - exact version - task category and constraints - observation count - independent source count - evidence class - freshness window - correction or dispute state - visible uncertainty
Relevant product: TrustMyChoice · Free REST + MCP lookup
Reproducible example
A famous server is evaluated for the wrong version
Server A has thousands of repository stars but recently changed its production interface. Server B is less well known but has three recent independent observations for the exact version and structured-text task the agent needs.
- Expected behavior
- Stars can keep Server A on the discovery list, but they do not substitute for version-bound runtime evidence. The comparison exposes source count, sample count, recency, task fit, and uncertainty. It does not declare Server B universally better or sell either provider a higher position.
- What it teaches
- A star average compresses context that an autonomous agent needs to make a bounded decision. The goal is not to abolish reputation; it is to stop reputation from masquerading as current execution evidence.
Production verification checklist
- Keep discovery popularity separate from task-specific outcome observations.
- Reject observations that cannot identify an exact endpoint, version, and task context.
- Show correction, dispute, sample-size, and recency state to both humans and agents.
Decision rule
No shared evidence system should silently convert insufficient observations into a confident score. Missing evidence must remain visible to the selecting agent.
Good fit
- You are comparing services whose performance varies by task or version.
- Agents need a machine-readable selection input.
- You want correction state and uncertainty to remain part of the record.
Use another approach when
- A verified security assessment, contract, or SLA is required.
- The decision is purely subjective and cannot be represented by bounded observations.
- You plan to purchase ranking position or manufacture contributions.
Options compared
| Option | Strength | Important limitation | Best fit |
|---|---|---|---|
| Star rating | Fast human summary | Context and uncertainty are compressed | Discovery and subjective sentiment |
| Directory placement | Finds available services | May reflect curation or sponsorship rather than outcomes | Candidate discovery |
| Benchmark | Repeatable controlled comparison | May not match production traffic | Capability and regression testing |
| Contextual evidence card | Task, version, sources, recency, and uncertainty | Cold start until real observations exist | Agent pre-selection |
Implementation cautions
- Never let payment alter evidence weight.
- Publish methodology and correction state.
- Keep prompts, outputs, credentials, and free-form customer content out of the evidence record.
Frequently asked questions
Are GitHub stars useless?
No. They can indicate awareness and community interest, but they do not measure task-specific runtime outcomes.
Why not produce one overall score?
A universal score hides task fit, version differences, sample size, and uncertainty.
How does the evidence network start?
Integrations contribute bounded structured observations from real use until independent-source publication thresholds are met.
How this guidance was reviewed
The product behavior described here is checked against ScrubMyText's public OpenAPI contract and automated contract tests. The worked example is synthetic and contains no customer data. Recommendations separate verified product behavior from broader implementation judgment, and every limitation remains visible rather than being converted into a marketing claim.
Related decisions
Next step
Use the product page for exact limitations and access requirements, then copy the corresponding REST or MCP workflow.