The routing decision
happens on-device,
before data leaves
Privacy-aware LLM routing. Shield classifies on your machine, a policy picks the backend, and only cleared or redacted content crosses the network. A library, not a hosted service — nothing of yours ever reaches our infrastructure.
See the decision, before anything is sent.
Three prompts, one policy. A privileged memo stays on a local model. A contract with names goes to the cloud — redacted first. A benign prompt goes to the cloud model you already use. Same code path; the sensitivity decides.
route() returns a decision, not a completion — the privacy-relevant choice is made, and auditable, before a single byte leaves the device.
“Draft a note about the privileged settlement memo.”
{"prompt_hash":"sha256:9f2c1a…","sensitivity_score":87,"route_decision":"ollama-local","rule_id":"privilege-stays-local","backend_is_local":true}Five guarantees a cloud gateway can't make.
A SaaS proxy routes on cost — after your content has already left the machine. These invariants are load-bearing, and they only hold because the decision runs in your process, before dispatch.
In-process, on your machine
Classification, policy, and redaction are library calls, not network calls. There is no Ogentic service in the path — no infrastructure that ever sees your content. You import it; it runs in your process.
Decision before dispatch
route() returns a RouteDecision — the choice of backend and why — not a completion. The privacy-relevant decision exists, and is recordable, before a single byte is sent anywhere. You dispatch second.
Local means local
The Ollama and llama.cpp adapters are loopback-only, enforced in code: they reject any non-loopback host. Content that stays local can only reach a model on your own machine.
Fail-closed
Content Shield flags as privilege, PHI, or MNPI can never resolve to a cloud backend — the router raises CloudRouteDeniedError before dispatch, even if a rule is misconfigured or mis-ordered. On by default.
Locality is authoritative when the config declares its backends — a misordered rule can't leak a privileged prompt to the cloud.
Shape-only audit
Every route() emits one decision row — a prompt fingerprint, the sensitivity score, category labels, and the chosen backend — never the raw prompt text. Error paths are recorded too. Fire-and-forget: a full or misconfigured log never crashes the router.
One YAML policy, read top to bottom.
Not a config console — a plain, versioned file. Each rule tests the on-device classification; the first match picks a backend and an optional transform. Anything that misses every rule falls back to default_backend.
version: 1
default_backend: ollama-local
deny_cloud: # fail-closed guard (on by default)
enforce: true
groups: [PRIVILEGE, PHI, MNPI]
budget: # per-call cost ceiling (on by default)
enforce: true
ceiling_usd: 1.00
rules: # first match wins
- id: privilege-stays-local
when: { groups_include: [PRIVILEGE, PHI, MNPI] }
route: ollama-local
- id: medium-redact-then-cloud
when: { sensitivity_score_gte: 30 }
route: openai-cloud
transform: shield_redact # mask before it leaves
- id: low-cloud
when: { sensitivity_score_gte: 0 }
route: openai-cloud- Match
- first rule that matches wins; else default_backend
- Groups
- groups_include / groups_exclude
- Score
- sensitivity_score_gte / _lt (0–100)
- Category
- category_in / category_not_in
- Transform
- shield_redact — mask before it leaves
- Guards
- budget ceiling · deny_cloud (fail-closed)
Guards run on every call — a per-call budget ceiling (default $1.00) and the fail-closed deny_cloud. Full reference: POLICY_REFERENCE ↗.
Wherever some content simply can't leave.
Regulated teams already run local models for the sensitive 10% and cloud models for the rest. The router makes that split a policy — automatic, auditable, and impossible to fat-finger.
Privileged matters stay on-device
Attorney-client privilege and work product never touch a cloud model. The privilege rule pins them to a local backend, and the fail-closed guard makes overriding it impossible.
PHI never leaves the building
Patient data is classified on-device and routed to a local model by policy. deny_cloud refuses a cloud backend even if a rule is misordered — HIPAA-shaped by construction.
MNPI stays inside the wall
Material non-public information is pinned local. Medium-sensitivity content gets shield_redact applied before any cloud call — the names are masked while the analysis still ships.
Drop-in for existing OpenAI code
Point base_url at the local router — one line. Benign prompts still reach the cloud model you already use; the privacy layer is invisible until the content actually matters.
Classify, decide, guard, dispatch.
Five steps from a prompt to a dispatched call. The one that changes everything is the second — the decision is made, and recorded, before the third step lets any byte leave.
classifyShield classifies
The prompt is scored on-device for privilege / PHI / MNPI and an overall sensitivity score. A library call — nothing has left the machine yet.
route()Policy picks a backend
First-match-wins rules choose a backend and an optional transform. route() returns the RouteDecision — the choice, made and recordable before dispatch.
guardBudget + deny-cloud
The estimated cost is checked against the per-call ceiling, and the fail-closed guard refuses any cloud route for privileged content. No partial sends.
dispatchSend the cleared payload
Only now is a backend called — a loopback-only local adapter, or a host-allowlisted cloud one — carrying only cleared or redacted content.
auditShape-only audit row
One JSON-lines row per call: a prompt hash, the score, the labels, the decision — never the raw text. Error paths included.
classify → route() → guard → dispatch → audit// route() returns a decision, not a completion — the privacy-relevant choice is made, and auditable, before a single byte is sent.
One async chat(), four backends.
Every adapter implements a single method — chat(messages, …) — and passes provider-native responses through unchanged. Two live on your machine (loopback-only); two reach the cloud (host-allowlisted). Swap where a route goes without touching the policy.
OllamaAdapter
Loopback-only — rejects any non-loopback host.
LlamaCppAdapter
Loopback-only; model pinned server-side.
OpenAIAdapter
Host-allowlisted; key read from env, never inlined.
AnthropicAdapter
Allowlisted; defaults max_tokens=1024.
The axis everyone competes on — inverted.
Every AI control plane routes on cost, latency, and quality, and decides after the content has left. That's the right tool for unregulated traffic. For regulated content, the axis and the timing are both wrong.
- Form factor
- SaaS proxy
- Routing axis
- Cost / latency / quality
- Where it decides
- Vendor servers
- Sensitive path
- Transits vendor + provider
- Audit
- Vendor dashboard
- Trust model
- Trust the vendor
- Form factor
- Local library
- Routing axis
- Sensitivity first, then cost
- Where it decides
- Your process
- Sensitive path
- Stays on-device
- Audit
- Shape-only, user-held
- Trust model
- Open source — verify it
Zero-data-retention is not local-first. “Deleted after we processed it” ≠ “never left the device.” A retention promise still ships your content to a third party who can be compelled to produce it while they hold it. Full sourcing: COMPARISON ↗.
Route a prompt in five lines.
The library makes the full per-prompt decision today; the OpenAI-shaped server is the drop-in for code you already have. Same policy, same classifier — Python 3.10+, Apache-2.0.
from ogentic_router import Policy, Router
policy = Policy.from_yaml("policy.yaml")
router = Router(policy) # default classifier: ogentic-shield
decision = router.route(
"Draft a note about the privileged settlement memo."
)
print(decision.backend_id) # -> ollama-local (privilege stays local)
print(decision.rule_id) # -> privilege-stays-local
print(decision.reasoning) # -> why this rule fired
# route() returns a decision, not a completion — you dispatch to
# decision.backend_id yourself, after the choice is already recordable.# 1. start the local router (one terminal)
# ROUTER_CONFIG=router.yaml ogentic-router serve # 127.0.0.1:8080
from openai import OpenAI
client = OpenAI(
base_url="http://127.0.0.1:8080/v1", # <- the only change
api_key="not-used-by-the-router", # server holds real keys
)
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Say hello."}],
)
# Inspect what the router will do: GET /v1/policyversion: 1
default_backend: ollama-local
deny_cloud: # fail-closed (on by default)
enforce: true
groups: [PRIVILEGE, PHI, MNPI]
rules: # first match wins
- id: privilege-stays-local
when: { groups_include: [PRIVILEGE, PHI, MNPI] }
route: ollama-local
- id: medium-redact-then-cloud
when: { sensitivity_score_gte: 30 }
route: openai-cloud
transform: shield_redact
- id: low-cloud
when: { sensitivity_score_gte: 0 }
route: openai-cloudpip install "ogentic-router[shield,cloud,local]" # route one prompt; print the decision (never calls a backend) ogentic-router route --policy policy.yaml \ --prompt "attorney work product" # the safe "what would happen" inspector — no server, no model call ogentic-router policies dry-run policy.yaml --prompt "..." # boot the OpenAI-shaped server (or the stdio MCP surface with --mcp) ogentic-router serve --config router.yaml --port 8080
Shield classifies. Router decides. Audit proves.
Composable Apache-2.0 primitives for regulated AI. Shield reads sensitivity; Router makes the on-device routing decision; Audit records tamper-evident evidence. Three are live today; the rest are on the way.
classify → route → prove · Router is the decision layer between ogentic-shield ↗ and ogentic-audit ↗ — part of the OgenticAI ↗ OSS suite.
Install it. Route in five lines.
The base install is slim — just the policy DSL. Add the extras you need: the classifier, the adapters, the drop-in server. No account, no telemetry, no service in the path.
pip install "ogentic-router[shield,cloud,local,server]"