Skip to content

AI Gateway↑

The AI Gateway is an on-premises forward proxy that sits between your AI applications and their LLM providers. It intercepts all LLM API traffic, applies your configured guardrail policies in real time, and forwards allowed traffic to the destination provider.

How it works↑

The AI Gateway operates as a transparent proxy:

  1. Your AI applications send requests to the AI Gateway instead of directly to the LLM provider.
  2. The gateway inspects the input message against your guardrail policies.
  3. If the message passes all checks, it is forwarded to the LLM provider.
  4. The LLM response is intercepted on the return path and inspected against output guardrails.
  5. The response is delivered to your application — either as-is, redacted, or blocked depending on the policy outcome.

Gateway components↑

The AI Gateway bundle includes three components:

  • App proxy (aispm-app-proxy): Forward proxy that routes incoming traffic and manages connections to LLM providers.
  • Guardrails service (aispm-guardrails): Runs detection models to evaluate messages against your configured policies.
  • Models: Downloadable ML models that power individual guardrail detectors (jailbreak, toxicity, etc.).

Prerequisites↑

Before deploying the AI Gateway, make sure you have the following:

  • An active AQtive Guard (AQG) instance with AI-SPM enabled.
  • Docker installed on the host machine where the gateway will run.
  • Network connectivity from the gateway host to your AQG instance and to your LLM providers.
  • Client credentials (Uploader role) for event reporting, and an API token for policy retrieval. Refer to Settings to create both.

Deployment↑

This section outlines the steps needed to deploy the AI gateway.

Download the gateway bundle↑

Download the AI Gateway bundle using your API token:

Bash
curl -H "Authorization: Bearer <your-api-token>" \
  -o ai-gateway-bundle.tar.gz \
  https://yourorg.aqtiveguard.com/api/v1/downloads/ai-gateway

Extract the bundle↑

Bash
tar -xzf ai-gateway-bundle.tar.gz
cd ai-gateway

The extracted bundle contains the container images, ML models downloader, configuration template, and the configure-gateway.sh management script.

Configure environment variables↑

Copy the .env template and fill in your values:

Bash
cp .env.template .env

Edit the .env file and set the following variables:

Event uploader↑

This connects the gateway to your AQG instance for reporting.

Tip

If you haven’t already, you’ll need to create a client credential with the Uploader role under Settings. Refer to Settings for details.

Variable Description Example
AQG_TOKEN_URL OAuth token endpoint for your AQG instance https://yourorg.aqtiveguard.com/authv2/realms/aqtiveguard/protocol/openid-connect/token
WEB_UPLOAD_URL Event upload endpoint https://yourorg.aqtiveguard.com/api/v1/events
AQG_CLIENT_ID Client credential ID aqg_client_...
AQG_CLIENT_SECRET Client credential secret aqg_secret_...

Policy source↑

This is where the gateway retrieves guardrail policies.

Variable Description Example
WEB_API_BASE_URL AQG web-api base URL https://yourorg.aqtiveguard.com/api/v1
WEB_API_KEY API token for policy retrieval <your-api-token>

If WEB_API_BASE_URL is left blank, the gateway loads policies from a local project_db.json file on the gateway host. This is useful for air-gapped environments or testing. Refer to AI guardrail policies for policy structure details.

LLM API keys (optional)↑

Configure these if the gateway proxies traffic to LLM providers on behalf of your applications:

Variable Description Example
OPENAI_API_KEY OpenAI API key sk-...
ANTHROPIC_API_KEY Anthropic API key sk-ant-...
GEMINI_API_KEY Google Gemini API key AIzaSy...

Initialize the gateway↑

Run the initialization script to load container images, download ML models, and start all services:

Bash
./configure-gateway.sh --init

This command:

  1. Loads the aispm-app-proxy and aispm-guardrails container images.
  2. Downloads the latest ML models for your configured guardrail detectors.
  3. Starts all gateway services.

Verify the gateway is running↑

Confirm the gateway is healthy:

Bash
curl http://localhost:10000/health

A successful response indicates the gateway is ready to receive traffic.

Note

The gateway serves HTTP by default. For production deployments, place a TLS-terminating reverse proxy in front of the gateway or configure TLS directly to ensure encrypted communication.

Management↑

Use configure-gateway.sh to manage the running gateway:

Command Description
./configure-gateway.sh --gateway-update Pull and apply updated container images.
./configure-gateway.sh --models-update Download the latest ML models.
./configure-gateway.sh --restart <service> Restart a specific service (e.g., aispm-app-proxy or aispm-guardrails).
./configure-gateway.sh --init --force Re-initialize the gateway from scratch (reload images, re-download models, restart all services).

Supported LLM providers↑

The AI Gateway supports proxying to LLM providers that expose an OpenAI-compatible API, including:

  • OpenAI
  • Google Gemini
  • Anthropic
  • Self-hosted models (for example, via Ollama)

AI applications can connect to the gateway using their native APIs. The gateway is designed to understand multiple existing API schemas and automatically handles the formatting differences between various providers.

Next steps↑

Once the AI Gateway is running, configure your AI applications to route traffic through it. Refer to Agent enrollment for instructions on updating your agents’ base URL, or deploy the Browser extension to protect browser-based AI interactions.

To customize the guardrails applied to traffic, see AI guardrail policies.