pacod-gw

Why a gateway

Every product that talks to a language model ends up re-implementing the same things: storing provider keys, translating between APIs, counting tokens, enforcing limits and explaining the bill. pacod-gw takes all of that out of the application. The application calls one endpoint with a gateway key; the gateway decides which model serves the call, what it may cost and who pays for it.

The first goal is concrete: replace a LiteLLM proxy in front of an application by changing only its base URL and key.

Two modes, one core

The core is one Rust binary with two listeners: the data plane for applications and the management API for the console and scripts. The mode is chosen by URL prefix and allowed per key.

• Proxy mode (/proxy/{provider}/…) relays the provider's own API byte for byte and adds authentication, limits and accounting. A template can still set the model and its parameters.
• Managed mode serves OpenAI's /v1/chat/completions, Anthropic's /v1/messages, Gemini's generateContent and the native /gw/v1/chat, plus embeddings and speech.

In managed mode the caller's dialect and the model's provider decide the route:

A call, step by step

1. Authenticate. The key is looked up by HMAC in an in-memory snapshot of the configuration.
2. Read. A scanner finds the model and stream flags without parsing the whole body; the workspace's plugins see the conversation.
3. Resolve. A template (@name or x-gw-template) or a granted model id picks the model and its parameters.
4. Admit. The key's concurrency cap and every applicable limit must pass.
5. Send. The stored provider credential replaces the gateway key. In managed mode a 429, a 5xx or a lost connection is retried, or moved to the next model, before the caller gets a byte.
6. Relay and settle. The answer streams back while a tap reads usage from a copy; then cost and counters are updated, and a usage event goes to a spool on disk.

The response carries x-gw-request-id, x-gw-delay-ms (the time the gateway itself added), x-gw-model-served and, for non-streaming managed calls, x-gw-cost.

Configuration lives in PostgreSQL, but the request path does not wait on it: a change swaps in a new snapshot within about two seconds, and if a store is down, traffic continues on the last one.

Organisations, workspaces and keys

An organisation is a customer or a large unit. A workspace is one application, project, environment or team, and owns its keys, templates, limits, settings and usage. Two rules cover every restriction: allow-lists intersect and numeric caps take the minimum.

Provider keys belong to the installation, an organisation or a workspace, and a call uses the nearest: the workspace's own, then its organisation's, then the installation's shared key. A key the gateway already stores can be passed down to an organisation or a workspace without entering it again; the secret never leaves the gateway.

Roles are one file, authz/gw.yaml, from which the core's and the console's checks are generated:

Calls carry attribution — end_user, session_id, trace_id, declared tags, agent_id, task, complexity — which reports, limits and routing read. People sign in to the console with local accounts or, with PGW_AUTH=oidc, through pacod-id.

Limits and cost

A limit applies to an organisation, workspace, key or end user. It counts tokens, requests or spend over a minute, hour, day, week, month or lifetime, and either refuses calls (429, 402) or only counts and reports. Applications can read a user's quota state from GET /gw/v1/quota. Counters are combined across instances through PostgreSQL about once a second, so a limit can be overshot by about a second of traffic.

Each call is priced when it settles, from a versioned price that covers cache reads and writes, reasoning, batch variants, tiers by input size and off-peak windows. There is no default price: a model needs a price, or an explicit "free", to be activated. Model discovery asks each provider for its model list, once a day by default, and proposes the models the gateway lacks; one added from a proposal stays a draft until it has a price. A scheduled price tracker reads LiteLLM's price list or a provider's pricing page and applies changes within a set bound, or proposes them for approval. While it has no sources, the console suggests LiteLLM's list for OpenAI, Anthropic, Gemini, DeepSeek and z.ai.

Routing and intent

A template can hold a pool of models with tiers, weights and priorities. Without further setup the first model serves and the next takes over on a 429, a 5xx or a lost connection. A routing scheme adds rules on what the caller states (complexity, task, tags, input size, or a CEL expression) and algorithms: priority, weighted, round robin, lowest latency and cheapest, with optional session affinity to keep prompt caches warm. A circuit breaker sets a failing model aside, and every response names the rule that routed it in x-gw-route.

The intent plugin answers "what is this call for?" against a catalogue the organisation or workspace writes, using an LLM, the Jev engine, or a cascade of both. Its verdict can block a call, route it, count it against per-intent limits or flag it for review; it can run in shadow mode first and never changes the conversation.

A new catalogue starts from a built-in template, starter, with a general assistant's intents and rules that flag and block. A new binding starts from one the gateway fills in for its scope: shadow mode, that catalogue, and the cheapest granted model that answers in a JSON schema, or Jev if there is none. The console's try and evaluate panels run the engines on sample texts, first on demonstration data made for the catalogue, so engines can be compared before a binding enforces. See the routing guide and the plugin concept.

Observability and audit

Usage events reach the analytics service in batches from each instance's spool; delivery is at least once, and ClickHouse removes duplicates. The service builds reports on usage, errors, gateway delay, rankings and end users, with CSV export, and runs the scheduled jobs. The console reaches it only through the core, which checks the caller's role and forces the scope.

Prometheus, Grafana dashboards, Alertmanager rules and OpenTelemetry traces cover operations; the console shows live figures, provider health and open alerts.

Audit mode keeps complete calls for debugging: the request, every provider attempt and the answers. It is off by default, an organisation can require or forbid it, credentials are removed first, and records expire after 14 days by default.

Security model

• Provider keys are sealed per credential with AES-256-GCM under a key-encryption key that comes from the environment or a file, never from the database. The API shows only a fingerprint.
• Gateway keys (pgw_…) carry 32 random bytes, are stored as HMAC-SHA256 with a pepper and shown once.
• Tenant separation. The workspace comes from the key, never from the request, and reports are scoped by the core to what the caller's roles allow.
• Outbound connections to providers and price sources are checked after name resolution: private and link-local addresses are refused unless an operator opens them.
• Prompts stay out of logs, metrics and traces, which carry metadata only; bodies are stored only in audit mode.

The threat model also lists what remains: the metrics port, for one, has no authentication and carries tenant labels, and on the shared network every pacod container can reach the management API, which then relies on its own authentication alone.

Deployment and operations

The core runs as all, or split into gateway instances and an admin instance. Instances share one PostgreSQL and nothing else.

Like every pacod project, the gateway joins the shared Docker network pacod-net, where each service has a name of its own: the core is pacod-gw, the other services gw-*. docker-compose.yml holds the gateway's own services; docker-compose.standalone.yml adds its copies of the shared components: PostgreSQL, pacod-id, Caddy and monitoring. pacod-demo runs the gateway together with the other pacod projects at demo.pacod.io. It loads only the first file and runs each shared component once; ClickHouse stays with the gateway.

Caddy terminates TLS and sends each call to the ready gateway with the fewest calls in flight. The runbook covers secrets, backup and restore, key rotation, upgrades and troubleshooting; a built-in tester runs test cases and load tests from the console.

pacod-gw-lite keeps translation, streaming, retries and failover, with per-key limits in memory and one log line per call with its cost. Its clients share the provider keys and are not kept apart at the provider. It is also pacod-gw on the shared network, so a host runs either it or the full gateway (details).

What has been tested

• OpenAI, Anthropic, Gemini, DeepSeek and GLM are checked with live calls; Kimi, Ollama, xAI and Vertex AI against their documentation and the mock provider only.
• The design target of 10,000 concurrent streams in proxy mode and 5,000 in managed mode per instance (2 CPUs, 1 GB) is not yet confirmed by a load test.
• Not yet exercised: Let's Encrypt certificates, a Linux host, an upgrade between releases and the Helm chart on a real cluster.
• Not in the first release: MCP and tool governance, content guardrails and response caching.

The architecture page shows how the code is laid out and where it differs from the design concept.