Open Source MCP Gateways: An Evidence-Based Evaluation Method
Evaluate an open source MCP gateway with reproducible protocol, isolation, authorization, bypass, failure and evidence tests instead of a feature checklist.
Evaluate an open source MCP gateway by running it against your protected actions and failure cases. Repository activity, supported transports and a long integration list matter, but they do not show whether the gateway rejects a wrong-audience token, isolates tenants, blocks a direct route or binds approval to exact tool arguments.
Start with the owner MCP gateway guide, then use this method to produce a review artifact that another engineer can reproduce.
Define pass criteria before choosing products
Weight criteria by the consequences of the tools behind the gateway.
| Area | Weight | Required evidence |
|---|---|---|
| protocol correctness | 15 | discovery, transport and error fixtures |
| identity and token isolation | 20 | audience, issuer, tenant and secret tests |
| action authorization | 20 | exact-argument policy and approval mutation tests |
| bypass resistance | 15 | network and upstream service identity test |
| operations | 15 | HA, upgrade, rollback and dependency outage drill |
| evidence | 10 | correlated decisions, calls and outcomes |
| maintainability | 5 | release process, security policy and dependency posture |
Do not award partial credit for a critical control that exists only in documentation. Record a test command, gateway version, configuration digest, expected output and observed output.
Use a portable harness contract
case_id: mcp-auth-wrong-audience
given:
gateway_version: "record-exact-version"
client: tenant-a-worker
token_audience: https://mcp.example.com/server-a
when:
server: server-b
method: tools/call
tool: issue_refund
arguments: { order_id: "9182", amount_minor: 4500 }
then:
gateway_status: denied
upstream_calls: 0
reason_code: TOKEN_AUDIENCE_MISMATCH
evidence_event: required
The MCP authorization specification requires HTTP clients to use protected-resource metadata and resource indicators, and requires servers to validate tokens intended for their resource. Make those normative requirements executable tests.
Test beyond the happy path
Include at least these families:
- malformed JSON-RPC, oversized messages and unknown methods;
- server metadata that points to an unapproved authorization server;
- two tenants using the same server name and tool name;
- tool descriptions that change after approval;
- injected identity headers and token passthrough attempts;
- duplicate calls under retries and lost responses;
- control-plane, replay-store and evidence-pipeline outages;
- upstream calls made around the gateway.
OWASP's agentic threats and mitigations supplies threat categories such as tool misuse, identity abuse and supply-chain compromise. Translate each relevant category into a gateway or upstream test.
Score claims and gaps separately
A comparison table should distinguish verified, documented, extension required, failed and not tested. "Supports OAuth" is not enough. Record which flow, discovery mechanism, audience behavior and token storage model you verified.
weighted score = sum(weight * verified_fraction)
critical gate = all mandatory cases pass
Selection requires both. A high score cannot compensate for a failed tenant-isolation gate.
Check license terms, release cadence and vulnerability handling, but avoid using popularity as a security proxy. A smaller project can pass the controls; a widely used one can fail your deployment model.
The MCP gateway security controls article provides detailed fixtures. MCP gateway versus API gateway explains when an existing API gateway needs MCP-aware extensions. For an implementation discussion, contact Intelliger with the protected tools and required failure behavior.
This evaluation method is product-neutral. Any comparison published by a vendor, including Intelliger, should disclose versions, weights, test limits and commercial relationships.
Evaluation questions teams miss
Should managed and self-hosted options use the same rubric?
Use the same control outcomes, then record who operates each dependency. A managed service must expose tenant isolation, regional processing, incident response, export and deletion evidence. A self-hosted project shifts patching, availability, signing keys and policy operations to your team. Do not award self-hosting an automatic security advantage or treat a managed control claim as verified without a test.
How should extension work affect the score?
Mark extension required separately from native and verified. Estimate the code, privileged integration points, upgrade burden and ownership needed to close the gap. An extension that intercepts every tool call may be feasible but becomes a critical security component. Add its tests and release cadence to the evaluation rather than crediting the underlying gateway for behavior it does not provide.
What product version belongs in the report?
Record exact release, edition, enabled plugins and configuration digest. Hosted services may change without a version visible to customers, so record the test date and observable capabilities. Re-run critical gates after upgrades. Avoid evergreen claims such as "Gateway X does not support OAuth" when the result came from one older edition or configuration.
Can documentation satisfy a criterion?
Documentation can establish intended behavior and guide a test. It should not satisfy a critical runtime criterion on its own. Mark the result documented, not verified if the team cannot execute the case. For operational claims such as regional failover or data deletion, request evidence or run a controlled exercise with the vendor.
How do teams compare usability without weakening security?
Measure time to onboard an approved server, diagnose a denial, rotate a credential, review metadata drift and export an incident package. A gateway that produces opaque errors will encourage bypasses. Usability scoring should reward clear recovery within policy, not one-click permission expansion or secrets pasted into client configuration.
When should a candidate be rejected regardless of score?
Define non-negotiable gates before testing. Cross-tenant access, token passthrough to the wrong server, an unblocked direct route, missing authorization on tool calls or unrecoverable duplicate writes can disqualify a candidate. Keep those results visible even if the weighted score is high because of routing and integration breadth.
Repeat the final shortlist tests with the people who will operate the gateway. Give them a wrong-audience denial, metadata drift, an upstream timeout and a suspected secret leak. Measure whether they can identify the affected tenant, server, tool and request without granting a wider scope or inspecting raw credentials. This is where a technically capable project can fail an enterprise evaluation: the control works, but its state and recovery path are too opaque for a pressured operator.
Publish the test repository or enough fixtures for another team to repeat the result. Screenshots of a dashboard are weak comparison evidence. Configuration, request, expected reason code and upstream-call count are much harder to misread.