AI / LLM Integration
Access LLM providers with managed prompts, optional retrieval over customer data, safety controls, cost guardrails, and fallback behavior when the upstream provider misbehaves.
Options
Provider Strategy*
Response Delivery*
Retrieval / RAG Strategy*
Governance Controls
Decision Points
Single provider or abstraction (LiteLLM, OpenRouter) for provider-swap?
If yes
Abstract via LiteLLM, OpenRouter, or Vercel AI SDK when you need fallback, multi-model routing, or want the option to swap. Accept you'll drop to raw SDK for bleeding-edge features.
If no
Single-provider direct SDK is the fastest path — use every native feature the day it ships. Plan the swap when lock-in becomes a real business risk, not before.
Self-host open-weight models, or managed APIs only?
If yes
Self-host when regulatory constraints prohibit third-party data sharing or inference volume makes in-house economics favorable. Budget for GPU ops as a first-class discipline — vLLM, sharding, autoscaling.
If no
Managed APIs are the pragmatic default. Frontier closed-source models still win on complex reasoning; infra time is better spent on product.
Stream responses to the UI or wait for full completion?
If yes
Stream any user-facing interaction — chat, generation, autocomplete. SSE is sufficient for unidirectional streaming. Perceived latency drops 10x; UX becomes acceptable on multi-second completions.
If no
Buffered is fine for background jobs, batch summarization, and structured-output tasks where the user isn't watching. Don't buffer anything in a live UI.
Persist prompts + responses for debugging/audit?
If yes
Log prompt, response, model, latency, tokens, and cost per call. Essential for debugging and compliance. Redact PII before write; enforce retention limits — prompt logs are sensitive.
If no
Running blind is a false economy — you cannot debug regressions without a log. At minimum sample-log (1-5%) with full fields.
Redact PII before sending to third-party providers?
If yes
Required for HIPAA, most enterprise deals, and EU deployments. Use Presidio, AWS Comprehend, or a redaction gateway. Validate detection on real samples — regex-only approaches miss too much.
If no
Acceptable only for products with no PII in prompts. Most products think they have no PII and are wrong — audit real traffic before deciding.
Cache responses for repeated prompts (semantic or exact)?
If yes
Exact-match caching (prompt hash + model + params) is a free 20-40% cost reduction on repetitive workloads. Semantic caching needs careful threshold tuning — too loose and users get wrong answers.
If no
Skip caching only if every prompt is genuinely unique (user-specific content, real-time data). Evaluate actual repetition before assuming it's zero.
Per-user / per-tenant token quotas?
If yes
Meter tokens in both directions per user and tenant. Alert at 80% of quota; hard-stop at 100% with a clear error. Without quotas, a loop, abuse, or prompt injection can 100x the monthly bill overnight.
If no
Skipping quotas is accepting unbounded downside. Even a flat free tier should have a safety ceiling.
Pass token cost through to customers or bundle into plan?
If yes
Usage-based pricing (per-token or per-call credits) aligns incentives but requires metering infrastructure, invoice lines, and customer-facing usage dashboards. Model margin carefully — token prices change.
If no
Bundling into a flat plan simplifies billing but exposes you to cost spikes. Set the quota ceiling so worst-case usage stays profitable.
Tool use / function calling in scope?
If yes
Design tools as a typed schema (JSON Schema or OpenAPI) with validation on model outputs. Limit tool blast radius — destructive tools require explicit user confirmation. Log every tool invocation.
If no
Pure text generation is simpler and safer. Add tools only when you have a clear agent use case, not speculatively.
RAG over customer data (embeddings + retrieval)?
If yes
Chunk strategy matters more than the vector DB choice — test 500 / 1000 / 2000 token chunks with overlap on real queries. Measure recall on a labeled set before scaling.
If no
Skip RAG if the model already knows the domain or queries are better served by structured API calls (tool use). RAG is oversold for transactional queries.
Vector store: pgvector, Pinecone, or managed service?
If yes
pgvector is the right default — your data is already in Postgres, no new infra. Graduate to Pinecone, Turbopuffer, or Weaviate past ~10M vectors or when pgvector indexing becomes a bottleneck.
If no
In-memory vector stores (FAISS, hnswlib) are fine for small static corpora loaded once per process. Avoid for anything that updates at runtime.
Version prompts like code (review, rollback)?
If yes
Store prompts in version control with code review, tag releases, and roll back via deploy. A prompt edit is a code change — treat it as such. Avoid runtime prompt stores unless you have an eval gate.
If no
Runtime prompt editing (via admin UI) is fast but dangerous — unreviewed edits ship instantly. Acceptable only with eval gating and instant rollback.
Safety / moderation filter on model outputs?
If yes
Pass outputs through a moderation classifier before rendering to end users. Defense in depth — never rely solely on the model's built-in refusals, especially for user-generated prompts.
If no
Skip only for closed internal tools with trusted inputs. Consumer apps always need output moderation.
Eval harness for prompt regressions in CI?
If yes
Maintain a golden set of 20-100 cases with expected outputs (or LLM-as-judge scoring). Run in CI on prompt changes and block regressions. The only reliable way to iterate on prompts at scale.
If no
Without evals, every prompt edit is a coin flip. Accept that most 'improvements' also cause regressions you won't notice.
Fallback model if primary rate-limits or errors?
If yes
Configure a secondary model (cheaper or different provider) to kick in on 429s and 5xxs. Degrade gracefully — surface a quality-degradation banner if the fallback is materially weaker.
If no
Single-model deployments take full outages when the provider hiccups. Acceptable only for non-critical paths with clear user messaging on failure.
Tradeoffs
Hard lock-in: an upstream outage becomes your outage, and migrating providers touches every prompt and SDK call
Perceived latency drops dramatically; infrastructure must handle long-lived connections (SSE/WebSocket)
Embedding costs, index storage, and retrieval latency on every query; recall quality is the new bottleneck, not model quality
Runaway costs from loops, bugs, or prompt injection are capped — quota enforcement pays for itself the first time it trips
GPU ops, model-update cadence, and inference reliability all become your responsibility
Implementation Examples
Direct access to Claude models with streaming, tool use, prompt caching, and extended thinking.
GPT-family models with streaming, function calling, vision, and the Assistants API for stateful agents.
Open-source provider abstraction with a unified OpenAI-shaped interface across 100+ models; fallback and budget controls built in.
Managed router across providers with unified billing, automatic failover, and a large model catalog.
Postgres extension for vector similarity search — the default RAG vector store for teams already on Postgres.
Managed vector database with metadata filtering, hybrid search, and multi-tenant namespace isolation.