What I check in an AI integration audit.

Before reading code, I ask for a one-page diagram showing how a user request becomes an LLM call and how the response gets back. Most teams cannot produce one. That is the audit’s first finding by itself: if the data flow is not written down, every engineer holds a slightly different mental model, and every refactor leaks complexity into the layer that happens to be open at the time.

The fix is small: write the diagram. The discipline is large: keep it current. The Phase 0 work the case study pages reference does this from day one — see the Paperbrief case study for what an ingest pipeline diagram looks like when it is real.

If the system is RAG, I ask for the recall numbers. Recall at k=5, k=10, k=20 on a labeled evaluation set. Most teams do not have a labeled evaluation set. They have vibes. The audit converts vibes into numbers, because the next time you change the embedding model, the chunking strategy, or the retriever, you need to know whether quality improved or regressed.

The smallest acceptable evaluation set is fifty real questions with one or two known-correct source chunks each. Hand-labeled. One afternoon of work. It pays back the first time someone says “let’s try the new embedding model.”

This is the failure mode that haunts every multi-tenant RAG system. A vector similarity search that forgets to filter by user returns chunks from every customer at once — and the LLM answer on top is plausibly grounded in someone else’s data. The query succeeds. The answer reads well. Nobody notices until they do.

I check whether isolation is application-level (every endpoint has to remember to add the filter) or structural (the database refuses to return cross-tenant rows because of row-level security plus a required filter argument on the similarity-search function). Application-level isolation is one forgotten WHERE clause from a breach. Structural isolation makes the dangerous path require intent. The Paperbrief case study walks through exactly how this is wired for one production system.

Prompts are software. If your prompts live as string literals scattered through the codebase, you cannot answer the question “did the AI get worse this week?” because you have no record of what changed. I look for a prompt registry: a named, versioned artifact with content history, owner, and ideally an evaluation harness that runs against a fixed set of test cases.

If the registry does not exist, the audit’s third finding writes itself. The fix is not glamorous — usually a JSON or Markdown directory checked into the repo, with a small loader. But the day you ship a prompt regression and have to roll it back, you will be very grateful you can.

Every AI integration I audit has made a vendor bet. Usually OpenAI. Sometimes Anthropic, Bedrock, or a local model. The bet is fine; the lock-in is the problem. I check whether the LLM provider sits behind a typed interface (a port, in ports-and-adapters terms) with the actual provider as one of one or more swappable implementations.

If the answer is no — if OpenAI’s SDK is called directly from 200 sites in the codebase — then the day the price-latency curve shifts, or a customer demands their data stays out of a specific provider’s hands, the work to migrate is structural rather than local. Adapter pattern is fifteen minutes of design and a lot of saved months.

I ask one question: how much did the most expensive customer cost you to serve last month? If the answer takes longer than five minutes to produce, the audit finds a cost-tracking gap. Per-call token usage — input, output, model — has to be recorded somewhere queryable, tied to the workspace or user who triggered it. Without this, you cannot price the product correctly, cap abusive users, or know whether a new feature is profitable.

The pattern I deploy is a single record_usage database call inside the same transaction as the request that consumed the tokens. One row in an events table, one upsert in a monthly rollup, no async drift. Billing becomes a SQL question, not a distributed-systems problem.

Cost tracking with no caps is a postmortem waiting to happen. I check whether there is a per-user daily token cap that returns a clear HTTP 429 when exceeded — and whether the cap is enforced before the expensive LLM call, not after. The audit usually finds either no cap at all, or a cap that fires after the work is done. Both are dangerous; the latter is worse because it gives a false sense of safety.

Median latency is a comforting lie. The P95 and P99 tell you what users actually feel. For chat, I check whether the response streams (it should) and what the time-to-first-token looks like at P95. For batch work — ingestion, evaluation — I check whether the long-running calls have a timeout and what happens when they hit it. The pattern that always loses: a job that retries forever because nobody set a budget.

Every LLM call should produce a log line with workspace ID, module, latency, input tokens, output tokens, cost, and the model used. Sampled prompt and response, ideally — redacted for sensitive content, but available enough to debug. Without this, the on-call engineer cannot answer “why did the AI start hallucinating last Tuesday?” without re-deriving the entire system from first principles.

The audit usually finds logging that captures latency but not tokens, or tokens but not workspace. Backfilling later is painful. The cost of doing this right on day one is one shared logging helper.

I read the runbook. If there is no runbook, I write the audit’s tenth finding before I read another line of code. An AI integration without a runbook means that when the provider is down, when a prompt has regressed, when usage suddenly spikes, the on-call engineer is improvising in front of an angry customer. The runbook does not need to be long — a one-page document with five common failure modes, how to detect each, and the first thing to try — but it has to exist.

The last thing I look at is the economics. I ask the team to multiply current usage by ten and tell me what the LLM bill looks like at that scale. If the answer is uncomfortable, the architecture has a cost trajectory problem — usually because retrieval is wasteful, the largest model is being used where a smaller one would do, or there is no caching of LLM responses for repeat queries. Cost economics that work at a hundred users and break at a thousand are the most common reason early-stage AI products quietly stall.