Questions to ask every vendor
Ask for each answer to be shown in the product, not just described. Where a vendor can't show it, treat it as not available for your evaluation.
Authentication
- Which identity providers are supported for SSO, and is SCIM provisioning included?
- Is each tool call tied to a named person and a named agent or client?
- How quickly does removing a user in the identity provider cut off their access?
Credential handling
- Can downstream calls use per-user OAuth tokens, and who completes consent?
- Where are refresh tokens and static secrets stored, and how are they encrypted?
- Is the client's token ever passed through to a downstream service?
Policy depth
- Can rules target a single tool, and specific arguments, per user or group?
- Can high-impact actions require human approval, and who approves?
- Can policies run in alert-only mode before blocking?
Audit
- Does every log entry include user, agent, tool, arguments and result?
- How are logs protected from tampering, and how long are they kept?
- Which SIEM or log destinations are supported for export?
Observability
- Does the gateway emit OpenTelemetry traces and metrics?
- Can we follow one agent session end to end?
- Is usage broken down by user, server and tool?
Deployment model
- Managed, self-hosted or both? In which regions is data stored?
- What does an upgrade involve, and do clients need reconfiguring?
- What compliance reports (for example SOC 2) can you share?
Shadow usage discovery
- Can you find MCP servers configured directly in clients on employee devices?
- Does discovery need employee opt-in, or does it work through our MDM?
- What happens when an unapproved server is found: alert, block, or a path onto the gateway?
A two-week proof-of-concept plan
Run the same plan against each finalist so the results compare.
- Before day 1. Agree success criteria with security and platform owners. Pick three MCP servers you really use: one read-only, one that writes, one that needs OAuth. Prepare a test tenant in your identity provider with two users in different groups.
- Days 1–3: connect. Deploy the gateway the way you would in production, using only public docs. Connect the three servers and two clients your teams use. Note every step that needed vendor help.
- Days 4–6: identity and credentials. Sign in as both users. Confirm each tool call is attributed correctly and that downstream calls act as the right person. Search configs, environment variables and logs for tokens. Deprovision one user and time the cut-off.
- Days 7–9: policy. Write a rule that lets one group read but not write on the same server. Add a human approval step for one destructive tool. Try to get around both from each client.
- Days 10–11: red team. Add a test server with an instruction hidden in a tool description. Plant an injected instruction in a ticket or document the agent will read. Attempt to send a dummy secret to an outside URL. Record what the gateway detected, blocked or logged.
- Days 12–13: audit, observability, shadow. Export the full log to your SIEM and reconcile it against what you did. Follow one session in your tracing tool. Add an MCP server directly to a managed laptop's client config and see whether it is found.
- Day 14: decide. Score the gateway on the seven criteria using what you observed, not what the slides said. Compare with the scores in the ranking and note any differences.
Use the criteria testsEach criterion on the criteria page has three concrete tests. They slot into the plan above.
Red flags
- Every user's calls show up under one service account in the downstream tool.
- Tokens appear in client config files or gateway logs.
- Policies apply per server only, with no way to block a single tool.
- Logs can't be exported, or don't name the person behind a call.
- The vendor can't say which MCP specification revision it implements.
- Deployment needs steps that aren't in the public docs.