On this page: 需求分层 · 路由策略 · 成本与SLA · FAQ · 转化
Agents, IDEs, and eval harnesses all want one OpenAI-shaped surface, but providers differ on timeouts, tools, and spend curves. Aggregation fixes the API mismatch yet shared queues still let batch traffic starve chat, weak cache rules leak prompts, and fallback ladders double bill. Here you get tiered requirements, a compact matrix on latency, concurrency, cache, and fallback, parameter placeholders, and a remote acceptance stance finance can trust. Read alongside OpenClaw plus LiteLLM proxy routing, the OpenTelemetry GenAI observability matrix, and the llama.cpp versus Ollama inference matrix before freezing production aliases.
1. One queue for humans and agents hides tail risk behind healthy averages.
2. Batch wins fade once memory bandwidth or cancelled tokens dominate without per-tier ceilings.
3. Fallback without marginal cost tags produces duplicate calls finance cannot reconcile.
需求分层
Name consumers first. Interactive chat needs tight first-token budgets, small per-session concurrency, and crisp error envelopes. Agents need higher inflight limits, tool-aware retries with fuses, and longer wall-clock tolerance. Offline eval or indexing favors throughput, wider micro-batches, and cheaper quant routes even if latency wiggles. Write one SLA blurb per tier covering max queue seconds, error budget, streaming needs, residency, and logging so cache policies stay lawful. Map tiers to hardware: laptops for spikes, staging clusters for integration, and dedicated Apple Silicon soak hosts because thermals and daemons change queues in ways laptops mask.
路由策略
Routing is aliases plus health plus breakers. Aliases decouple clients from vendor renames. Health should blend timeout ratios, token ceiling breaches, and memory pressure—not only pings. Breakers open per alias and tenant so noisy workspaces cannot brown out neighbors. Document when sessions stick to a warm KV host versus round robin stateless nodes.
| Pattern | Latency posture | Concurrency posture | Cache posture | Fallback posture |
|---|---|---|---|---|
| Direct vendor HTTP | Minimal hops; fragile regional tails. | Sudden per-key throttles; weak fairness. | Mostly vendor-side; little dedupe. | Manual reroutes; dual-spend risk. |
| Edge API gateway | Slight added latency; calmer TLS churn. | Central quotas; watch partition hotspots. | Good for idempotent reads; risky for chat bodies. | Policy reroutes need per-hop cost tags. |
| OpenAI-compatible aggregation | Small parse tax; gains from batching and locality. | Fairer across aliases; needs tier pools. | Template caches and KV hints if policy allows. | Budgeted downgrade ladders; audit each hop. |
| On-device Metal stack | Great local payloads; remote tools add jitter. | Unified memory caps; bursty single tenant friendly. | Hot in-process reuse; weak cross-host reuse. | Spill to cloud alias past RAM envelopes. |
Use the table in reviews: pick the two loudest columns, attach last sprint metrics, and assign a default pattern per tier.
成本与SLA
Cost is tokens plus queue seconds plus rework from bad completions. SLAs should cite p95 first-token latency, consecutive failures before breakers open, and recovery time when a region browns out. Dashboards must slice spend by alias, tenant, and fallback depth so invoices reconcile without opening traces. Remote math should add hourly rent, egress, and idle minutes—not only API list prices.
Citable checklist snippets you can paste into runbooks:
- Every alias lists provider, region, quantization, and dollars per million tokens for the active rate card.
- Batch jobs publish micro-batch windows and cancel rules when clients disconnect.
- Fallback emits reason codes plus estimated marginal cost before the alternate model runs.
Parameter placeholders belong in secret stores, not git history.
# Gateway and pooling
OPENAI_BASE_URL=${AGGREGATION_BASE_URL}
ROUTING_TIER_INTERACTIVE_MAX_INFLIGHT=${ROUTING_TIER_INTERACTIVE_MAX_INFLIGHT}
ROUTING_TIER_AGENT_MAX_INFLIGHT=${ROUTING_TIER_AGENT_MAX_INFLIGHT}
ROUTING_TIER_BATCH_MAX_INFLIGHT=${ROUTING_TIER_BATCH_MAX_INFLIGHT}
# Batch and streaming
COMPLETION_MICRO_BATCH_MS=${COMPLETION_MICRO_BATCH_MS}
COMPLETION_MAX_BATCH_TOKENS=${COMPLETION_MAX_BATCH_TOKENS}
STREAM_CHUNK_BYTES=${STREAM_CHUNK_BYTES}
# Cache and fallback
PROMPT_CACHE_MODE=${PROMPT_CACHE_MODE}
KV_CACHE_REUSE_POLICY=${KV_CACHE_REUSE_POLICY}
FALLBACK_MODEL_ALIAS_CHAIN=${FALLBACK_MODEL_ALIAS_CHAIN}
FALLBACK_MAX_EXTRA_SPEND_USD=${FALLBACK_MAX_EXTRA_SPEND_USD}
# Remote acceptance host
REMOTE_MAC_SOAK_HOURS=${REMOTE_MAC_SOAK_HOURS}
REMOTE_MAC_NOTARIZED_CHECKLIST_ID=${REMOTE_MAC_NOTARIZED_CHECKLIST_ID}1. Inventory clients and freeze OpenAI routes, streaming flags, and tool formats.
2. Ship per-tier concurrency pools and timeouts, then mixed-traffic load tests.
3. Emit traces with alias, fallback depth, cache hits, and queue seconds before broad onboarding.
4. Canary on staging hardware until breaker drills page the real on-call rotation.
5. Replay hours of traffic on a rented remote Mac, compare p95 and p99 to laptop baselines, and archive dashboards plus finance sign-off.
FAQ
Should agents and humans share one routing table? Split them. Dedicated pools, retries, and failure envelopes stop tool loops from draining chat concurrency.
Does a larger batch always reduce cost? No—watch cancelled tokens, memory pressure, and prefill queues before trusting cheaper averages.
Why validate routing on a remote Mac instead of a laptop? Stable power and networking remove sleep jitter so acceptance charts match how you will host long-lived gateways.
转化
Credibility gates rollout. A dedicated Mac mini M4 cloud node pins gateways, replays traces, and freezes SLA charts before aliases widen. Browse purchase and pricing without signing in, skim Help Center runbooks, and keep digging via the Tech Blog index plus the LangGraph checkpoint and sandbox guide when state shares the same router.
Public pages: Pricing, Purchase, and Help Center are readable without login; the Tech Blog lists related routing and observability guides.