What is the Cato AI GW

Prev Next
This content is currently unavailable in French. You are viewing the default (English) version.

Overview

Cato AI Gateway lets you secure, control, route, and monitor traffic between AI-powered applications and the AI models they use. It provides a Cato-managed path between your AI applications or users and their configured model providers, so you can consistently apply security controls, identify application context, and maintain an audit trail for AI interactions.


Without AI Gateways, when each AI application or user connects to a model provider independently, it is difficult to identify which application generated a request, apply the right security policies, and maintain a consistent audit trail across your AI environment. With the Cato AI Gateway, Cato authenticates the application, identifies its context, applies the relevant security controls, and then forwards permitted traffic to the configured model provider. The Cato Management Application (CMA) is the single console to configure AI Gateway components, create policies, and monitor traffic.


The Cato AI Gateway is built around these core components:

  • Gateway Key that authenticates the application,

  • Destination that the key connects to (a single Model Provider directly, or a Virtual Model that can route across several providers)

  • Enforcement policy that applies AI Security controls

Control Plane vs Data Plane

Cato AI Gateway separates the control plane from the data plane. Routing, authentication, and enforcement are all defined in the control plane, which has no access to the AI traffic itself. That traffic, every prompt and response moving between an application and a model provider, flows through a separate data plane instead.

By default, the data plane is Cato's Cloud. If you need to meet data residency requirements, you can use an on-prem Outpost deployment instead.

Use Case: Protect an Internal AI Assistant

Company ABC uses an internal AI assistant to help employees search internal documentation, summarize content, and answer operational questions. The assistant connects to its model provider independently, the same way every other AI integration in the company handles its provider connection. The IT team has no consistent way to identify which application generated a given request, apply the right security policy to it, or maintain an audit trail across AI interactions like this one.
After deploying the Cato AI Gateway:

  1. The AI assistant sends its requests through Cato, authenticating with the dedicated Gateway Key, mapped to a Homegrown Agent that identifies it.

  2. IT creates an enforcement rule scoped to that agent, applying an Engine Profile that checks for sensitive data and unsafe content, with an action to anonymize or block matches.

  3. Cato can inspect prompts and, where supported, responses accordingly, allowing compliant interactions through and blocking or logging unsafe ones before they reach the model provider or return to the employee.

How Cato AI Gateway Works

Cato AI Gateway sits inline between your AI agents or users and the AI provider. A request follows this path:

  1. An AI agent or user uses the Gateway Key to send a request to the Model Provider via the Cato AI Gateway.

  2. Cato authenticates the key and identifies the Homegrown Agent or user context.

  3. The request is evaluated against any enforcement rule scoped to that identity, applying the assigned Engine Profile and action.

  4. If the request is permitted, Cato forwards based on the model access:

    1. Direct Model Access: Cato forwards the request to the configured Model Provider

    2. Dynamic Model Routing: Cato forwards the request to a target selected by the Virtual Model's routing strategy

  5. Cato records the interaction and evaluates the response before sending it to the application.

Capabilities of the Cato AI Gateway

Connect to an AI Model

A Model Provider represents the connection Cato AI Gateway uses to reach a model vendor's API. You choose how the vendor's credential reaches Cato:

  • Store it directly in the provider configuration so Cato manages it centrally

  • Each application supplies it via a request header instead, this keeps the credentials with the calling application and avoids storing the data with Cato

The connection can be hosted in the Cato Cloud, or an Outpost deployment if the traffic needs to stay inside your own network. For vendors that support regional processing, you can pin the connection to a specific endpoint to match your data residency setup.
Supported vendors include: OpenAI, Anthropic, Google Gemini, Microsoft Foundry, Amazon Bedrock, Azure OpenAI Service, or a custom provider.

Data Residency

For providers like OpenAI, you can pin the connection to a specific processing region (for example, us.api.openai.com or eu.api.openai.com) instead of the default global endpoint. This lets you match the Gateway's provider traffic to data residency commitments already configured on your provider account.

Deployment Flexibility

Each Model Provider is hosted on the Cato Cloud or on a customer-managed outpost. Outposts let you keep traffic inside your network and avoid exposing internal LLM deployments to inbound connections from the Internet.

Route Requests Across Multiple Models

A virtual model is an abstraction layer a Gateway Key can target instead of a direct connection to a single Model Provider. It presents one consistent API schema to applications, then routes each request to one of several underlying models, potentially spanning multiple Model Providers and vendors. This provides resilience for applications to different model providers.
Requests to a Virtual Model follow a priority order you define. Cato sends the request to the first target on the list. If that target fails, it automatically retries the next one, and so on down the list. In addition, the virtual model spreads requests evenly across targets, splitting them by weight, or routing based on which target answers fastest. Capabilities include:

  • Health checks to detect when a provider or model target is unavailable

  • Failover to a backup provider or model, including a configurable delay rather than waiting for a full timeout, and logic to avoid flapping between targets

  • Load balancing across multiple provider or model targets, potentially including sub-groups of the same provider (for example, spreading load across multiple Azure subscriptions)

Authenticate to the AI Gateway

A Gateway Key is the credential an AI application uses to authenticate to the Cato AI Gateway. The application includes its Gateway Key when sending a request. The key identifies the application's traffic to Cato and associates it with its configured application context and Model Provider, giving admins a clear way to distinguish traffic from different AI applications, services, or agents.
For example, an internal employee assistant and a customer-facing support chatbot can each use a different Gateway Key, letting you manage, monitor, and investigate their AI activity independently. If a key is exposed, retired, or no longer needed, you can rotate or remove it without changing how other applications use the gateway.
When you create a Gateway Key, configure how it reaches the provider:

  • Direct Model Access: the key uses any model supported by a single configured provider. Routing features such as failover and load balancing aren't available in this mode.

  • Dynamic Model Routing: the key uses a virtual model that can resolve to one or more providers, adding routing and provider abstraction. This mode is marked Coming soon and isn't available yet.

Homegrown Agents Represent your Apps

Mapping a Gateway Key to a Homegrown Agent establishes the application context Cato uses to identify that key's traffic. This mapping is optional at creation time and can be set later, but it's what determines which Guard protects the traffic on that key: the Guard applied to a request is the one associated with the Gateway Key's mapped agent, not something you pick separately on the key itself.

Applying a Security Policy

The Guards Policy applies AI Security controls to native AI Gateway traffic. Each rule is for a Homegrown Agent or User and designates which Engine Policy to apply to the inspected traffic. Different actions are applied to matching traffic, such as Monitor, Block, or Block and Anonymize. This helps you protect sensitive information and enforce AI usage policies for AI applications you build without disrupting legitimate business use.

Visibility for AI Cost and Usage

The AI Gateway Overview page in the CMA helps you track how AI Gateway traffic is used over time. IT shows which applications or people are driving usage, which models are consuming the most tokens, and whether usage is trending up or spiking unexpectedly. You can catch runaway or unusual activity before it becomes a cost or security problem.