Model gateway (LiteLLM)

The model gateway is optional, and most installs start without it. Without the gateway, users bring their own agent subscriptions: they sign in to Claude or Codex directly from the desktop app, exactly as they would on their own machine. Nothing about the base install requires the gateway.

Add the gateway when you want, as an admin:

• Central provider keys. Your Anthropic, OpenAI, or xAI keys live in one place (the LiteLLM proxy container), not on laptops and never inside sandboxes.
• Per-user virtual keys. The Proliferate server mints and manages a durable virtual key for each user; agents and sandboxes only ever see the public gateway URL and that virtual key, never your real provider credentials. The server rotates keys itself; there is nothing for an operator to expire or renew by hand.
• Budgets. Spending limits per user and per organization, enforced at the gateway.

How it works

The gateway is LiteLLM, and the standard Docker Compose stack can run it for you: litellm and its own Postgres database (litellm-db) ship as an optional service group in docker-compose.production.yml, off by default and turned on automatically by one flag — AGENT_GATEWAY_ENABLED=true. bootstrap.sh and update.sh detect the flag, pull/start litellm and litellm-db under the agent-gateway Compose profile, and wait for litellm to report healthy before handing back control; there is no separate profile command to run. The bundled Caddyfile also always reserves a /llm path on your existing SITE_ADDRESS for the gateway's public traffic, so there is no second hostname or Caddy edit either — you just point AGENT_GATEWAY_LITELLM_PUBLIC_BASE_URL at it.

The image is Proliferate's own build (ghcr.io/proliferate-ai/proliferate-litellm), preloaded with the model list Proliferate's harnesses expect (current Claude, GPT, and Grok models, each mapped to your own provider key). That model list is baked into the image at build time, not a file you edit on the box — a different or expanded model list ships as a new proliferate-litellm image release, not an on-host config change. You supply provider keys and a master key; the Proliferate server uses the master key to mint and revoke per-user virtual keys, and sandboxes call the gateway's public URL with those virtual keys. The master key itself stays between the Proliferate server and LiteLLM, and is never sent to sandboxes.

Prefer to run LiteLLM yourself (a different host, a managed instance, or a custom model list) instead of the bundled service? See external LiteLLM instead below. The Proliferate server only cares about the base URL, public URL, and master key; it does not care who runs the process behind them.

Set up

AGENT_GATEWAY_ENABLED=true
 
# The URL agents/sandboxes use to reach the gateway. handle_path strips the
# /llm prefix, so this must be https://<SITE_ADDRESS>/llm, not a bare host.
AGENT_GATEWAY_LITELLM_PUBLIC_BASE_URL=https://proliferate.company.com/llm
 
# Internal Docker network address of the bundled service (already correct
# for the bundled service; only change this for external mode, below).
AGENT_GATEWAY_LITELLM_BASE_URL=http://litellm:4000
 
# LiteLLM's own master key, and Postgres password for its key/spend store.
# Generate both with `openssl rand -hex 32`.
LITELLM_MASTER_KEY=sk-<random-value>
LITELLM_POSTGRES_PASSWORD=<random-value>
 
# The Proliferate server authenticates to LiteLLM's admin API with this; it
# must be the exact same value as LITELLM_MASTER_KEY above.
AGENT_GATEWAY_LITELLM_MASTER_KEY=sk-<same-random-value>
 
# Provider keys the bundled LiteLLM container calls out with. Set the ones
# your team uses; an unset key just means that provider's models 400.
ANTHROPIC_API_KEY=sk-ant-...
OPENAI_API_KEY=sk-...
XAI_API_KEY=xai-...

./update.sh   # or ./bootstrap.sh on a fresh install

curl -fsS https://proliferate.company.com/llm/health/liveliness

Environment variable reference

Bundled-service-only (skip these in external mode):

Budgets

AGENT_GATEWAY_DEFAULT_USER_BUDGET_USD (default $5) caps what a single user can spend through their personal virtual key. Once a shared sandbox or automation run authenticates as the organization, that traffic is capped separately by AGENT_GATEWAY_DEFAULT_ORG_BUDGET_USD (default 0, meaning uncapped). These are lifetime caps with no automatic reset: when a key hits its budget, the gateway rejects further model calls until an admin raises that key's budget in LiteLLM directly. Provider spend itself is whatever your provider accounts bill; LiteLLM's own logs are the place to watch usage per key.

The bundled model list

The bundled image ships a fixed, checked-in model list mapping model names (current Claude, GPT, and Grok releases) to your provider keys; it is not a general-purpose "add any model" proxy. That covers every model Proliferate's supported harnesses ask for. If you need a different provider (Azure OpenAI, Bedrock, a self-hosted model server) or a custom model list, run LiteLLM yourself instead.

External LiteLLM instead

If you'd rather run LiteLLM outside the bundled Compose services, skip the agent-gateway profile entirely and point the same three AGENT_GATEWAY_LITELLM_* variables at your own instance: its admin URL, its public URL, and its master key. The Proliferate server does not distinguish between the bundled container and an externally run one; it just calls whatever URLs you configure. This is the path for a custom model list, a managed LiteLLM service, or a LiteLLM instance you already run for other purposes.

Removing the gateway

Set AGENT_GATEWAY_ENABLED=false, run ./update.sh, and if you were running the bundled service, stop it too:

docker compose --env-file .env.runtime -f docker-compose.production.yml \
  --profile agent-gateway down

Users fall back to signing in to their agents directly with their own subscriptions, the same as an install that never enabled the gateway.