A criterion scores 2 when it is built in and documented, 1 when it is present with gaps, and 0 when we couldn't find it in the vendor's public documentation. The methodology explains the rest.
1. Authentication
Does the gateway know which person and which agent made each MCP request?
- Strong Users sign in through your identity provider (SSO), groups sync from it, and every tool call carries a verified user and agent identity.
- Partial The gateway checks a token or API key, but identity is per client or per service account rather than per person.
- Not documented We could not find it in the public docs.
How to test it
- Connect the gateway to a test tenant of your identity provider and sign in as two different users.
- Make the same tool call as each user and confirm the gateway's logs name the right person, not a shared key.
- Remove one user from the identity provider and confirm their next call is refused.
Strong today: Runlayer, Obot, agentgateway, Kong AI Gateway.
2. Credential handling
Can agents use downstream tools without ever holding long-lived secrets themselves?
- Strong The gateway runs the MCP OAuth flow, stores per-user tokens or issues short-lived credentials, and injects them at call time so clients never see them.
- Partial Secrets are kept out of client config, but they are shared across users or managed only on one machine.
- Not documented We could not find it in the public docs.
How to test it
- Connect an OAuth-protected MCP server (for example GitHub or a ticketing tool) through the gateway as two users.
- Check that each user's calls act as that user in the downstream tool's own audit log.
- Look for the token in client config files, environment variables and gateway logs; it should be in none of them.
3. Policy depth
Can you allow or block individual tools, arguments and actions per user or group, not just whole servers?
- Strong Rules apply at tool-call level, by user, group, client and server, with options to require human approval for risky actions.
- Partial Access is controlled per server or per route, or rules exist but need custom code or plugins to reach tool level.
- Not documented We could not find it in the public docs.
How to test it
- Write a rule that lets one group read from a server but not call its write or delete tools.
- Call the blocked tool as a member of that group and confirm the gateway refuses it with a clear reason.
- Change the rule and time how long it takes to apply without restarting clients.
Strong today: Runlayer, Obot, agentgateway, Kong AI Gateway.
4. Audit
Is there a complete, attributable and exportable record of every tool call?
- Strong Every call is logged with user, agent, tool, arguments and outcome, kept tamper-resistant and exportable to your SIEM.
- Partial Calls are logged, but without full attribution, retention controls or an export path you would trust in an investigation.
- Not documented We could not find it in the public docs.
How to test it
- Run ten tool calls across two users, then export the log and check every call is there with user, tool and result.
- Ask the vendor how logs are protected from edits and how long they are kept.
- Send the logs to your SIEM or log store and confirm the fields arrive intact.
5. Observability
Can engineers trace a session end to end, with latency, errors and usage, in the tools they already use?
- Strong OpenTelemetry traces and metrics, session-level views, and usage broken down by user, tool and server.
- Partial Logs and basic metrics exist, but no standard tracing or session view.
- Not documented We could not find it in the public docs.
How to test it
- Point the gateway's OpenTelemetry output at your collector and follow one agent session from request to tool result.
- Force a downstream error and confirm it shows up with enough context to debug.
- Check whether usage can be broken down per user and per server.
Strong today: Runlayer, agentgateway, Kong AI Gateway, IBM ContextForge.
6. Deployment model
Can it run where your policies require (managed, self-hosted, local), and how much work is it to operate?
- Strong A documented deployment path that fits production: managed service, self-hosted on Kubernetes, or both, with a clear upgrade story.
- Partial Runs in one environment only (a developer laptop, one cloud), or self-hosting options are not stated publicly.
- Not documented We could not find it in the public docs.
How to test it
- Deploy it in the environment you would use in production, following only the public docs, and note every step that needed help.
- Confirm where data (logs, tokens) is stored and in which region.
- Run an upgrade and check whether clients had to reconnect or be reconfigured.
Strong today: Obot, agentgateway, Kong AI Gateway, IBM ContextForge, Docker MCP Gateway, Lasso MCP Gateway, Microsoft MCP Gateway.
7. Shadow usage discovery
Can it find MCP servers and AI clients people set up on their own, outside the gateway?
- Strong Discovers unmanaged MCP servers and client configs on employee devices (for example through MDM) and helps move them onto the gateway.
- Partial Some device-level inventory exists, but in beta or limited to certain clients.
- Not documented We could not find it in the public docs.
How to test it
- On a managed test laptop, add an MCP server directly to a client's config without going through the gateway.
- Check whether the gateway reports it, how long that takes, and whether it needed the employee to opt in.
- See what the product offers next: block, alert, or a path to bring that server under policy.
Strong today: Runlayer.
What we don't score
Price, performance benchmarks and customer ratings are not part of the score. Prices are mostly unpublished, and we have not run controlled benchmarks, so we don't quote numbers we can't stand behind. Each gateway's page says where to check its current price.