Why AI audit agents need more than search results

I asked an AI agent to build a patient review list from 1000 synthetic patient records. It returned 3 patients. All 3 belonged on the list. So did 9 others that it never mentioned.

If I had checked only the 3 names it gave me, the answer would have looked right. I knew it was incomplete because I had built the test records myself and written down the expected answer before I ran anything.

This is Part 2 of Beyond Similarity. Part 1 used a grocery substitution to show why AI agents need clear business definitions. This post applies the same idea to an audit: write down who belongs on the list, then check whether the agent finds all of them.

Search finds something. An audit needs everyone.

Search is built to return the few records that look most like your question. An audit asks a harder question: did we find everyone? A list with missing names fails quietly. Nothing on it is wrong, so nobody notices what is absent. The only way to catch the gap is to know the right answer in advance and compare.

The question I asked

In plain words: find the patients with type 2 diabetes who were admitted to hospital recently, whose kidney function had dropped below a set level, and who were then prescribed 1 of 4 named diabetes drugs.

Kidney function here is measured by eGFR, a lab result that estimates how well the kidneys filter blood. A lower number means weaker kidneys. For this test, a reading below 45 was the flag for review. That is a simplified test rule, not a clinical judgement.

Here is the full question as I wrote it:

As of the audit date, find all patients with an active type 2 diabetes diagnosis whose inpatient hospital admission started in the last 30 days and whose most recent eGFR reading is below 45 mL/min/1.73 m². Include only patients with an active prescription written after that reading for metformin, sitagliptin, linagliptin or alogliptin, including brand-name and combination products.

So a patient had to meet all 4 conditions as of the audit date:

1. an active type 2 diabetes diagnosis
2. a hospital admission that started in the last 30 days
3. a most recent eGFR reading below 45
4. an active prescription for a product containing 1 of the 4 named drugs, written after that reading

The order of events matters. The prescription has to come after the low reading, and that reading has to be the latest one on file.

The test data

I built 1000 synthetic patients with diagnoses, admissions, prescriptions, lab results and some clinical notes. No real patient appears anywhere in this post.

Of those, 12 were built to match the rule: 7 with a prescription that named metformin, and 5 with a prescription under a brand name only, such as Glucophage or Riomet. The word “metformin” appears nowhere in those 5 records.

Another 38 were near misses. Each looked like a match and failed exactly 1 condition: an admission too old, a prescription that had been stopped, a reading just above 45. The other 950 could not match.

Because I planted the answers first, I could score any list the system produced: how many of the 12 it found, and how many names it added that should not be there.

3 ways to build the list

Search, then write. This is the common setup called retrieval-augmented generation, or RAG. The records are split into chunks, small pieces of text that search can index. The agent searches for the chunks that look most related to the question, hands the best few to a language model, and the model writes the answer from those chunks only. In my setup the search covered 1334 chunks: every clinical note, plus a text version of each patient’s structured record. It ranked them by word overlap and by similarity in meaning, then passed the top 8 to the model.

Let the model write the query. The second approach skips search. It gives a language model the database table definitions and a few sample rows, and asks it to write the query itself. The model has to work out from the question which drugs count, which reading counts and how the dates relate.

Check every record against the rule. The third approach is a query written from the 4 conditions above and run against every patient. For the drug condition it uses a published list of which brand-name products contain which ingredient, so a prescription for “Glucophage” counts as metformin even though the word never appears in the record.

What happened

Result	Search + model	Model-written query	Rule-based query
Known matches found	3 of 12	7 of 12	12 of 12
Known matches missed	9	5	0
Names added that failed the rule	0	14	0
Brand-name patients found	1 of 5	0 of 5	5 of 5

The search answer had no wrong names. That is what made it convincing. The 9 missing names were the problem. The model-written query had the opposite problem: it found more of the right patients, and it added 14 names that did not belong.

These are single runs on synthetic data. They describe this setup and say nothing about how other systems perform.

Why search stopped at 3

Follow the search approach in order: the question goes to search, search picks the 8 chunks that look most like it, and the model writes from those 8. A patient can be in the records and still never reach the model.

Raising the limit does not fix this. It is tempting to read the result as a settings problem. I asked for 8 chunks, there were 12 patients, so ask for more. I did not test a larger limit, and it might have found a few more. But the number 8 is not the real issue. An audit asks how many patients meet the rule, and you do not know that number until the audit is done. So you cannot know how many chunks to ask for. Ask for 20 and you may still be short. Ask for 100 and the model is reading mostly records that do not belong on the list, with nothing to tell it when it has them all. Search ranks chunks by how closely they resemble the question. It never says “that is everyone”.

Resembling the question is not the same as meeting the rule. Even inside the 8 slots, only 3 went to patients who belonged on the list. The model added no wrong names, so the other 5 slots went to records that looked right but failed a condition, or to a second chunk about a patient it already had. A clinical note about declining kidney function can outrank the structured record that holds the actual lab value and dates. Search measures how much a chunk is about the topic. The rule asks whether 4 specific conditions are all true at once. Those are different questions.

Brand names share no words with the question. “Glucophage XR 500 MG” says nothing about metformin. Search found 1 of the 5 brand-name patients. It also missed 5 of the 7 generic-name patients, so brand naming is not the whole story.

I could see the mechanism in an earlier version that searched only the clinical notes. It found 4 of 12, and none of the 8 patients it missed appeared in the chunks handed to the model. The model cannot name a patient it never saw.

That is the check to run on your own system. For each name you expected, ask whether it reached the model. If not, search left it out. If it did, and the model still dropped it, the problem is in the answer stage. The two need different fixes.

Why not just let the model write the query?

That was my first attempt, and it is the obvious answer to the search problem: skip the chunks, give the model the tables and let it query everything. It returned 21 patients. Of those, 7 belonged on the list and 14 did not. Of the 5 brand-name patients, it found none.

So the model’s reading of the rule was wrong in both directions. It was too loose somewhere, because 14 patients who failed a condition got in. It was too narrow somewhere else, because 5 matches were left out, and every brand-name patient was among them. The query ran against every record, which is what search could not do, and it still produced a list a reviewer could not trust.

The difference between this query and the one that scored 12 of 12 is not the database. The second query was written from conditions someone had spelled out, with the brand-to-ingredient links taken from a published source, and it was checked against known answers before anyone read its output. A model can draft that query. The draft is a guess at what the rule means, and the guess has to be checked.

A single patient, step by step

Take pat-000043, a synthetic patient who belongs on the list. The audit date is fixed in the test data as August 13, 2026, so “the last 30 days” means July 14 to August 13. The records show:

• an active type 2 diabetes diagnosis
• a hospital admission on July 25, inside the 30-day window
• a latest eGFR of 36.8 on July 26, below 45
• an active prescription written on July 29, after the lab result, for “Glucophage XR 500 MG 24HR Extended Release Oral Tablet”

The dates and status checks pass. The last step is the drug. The query needs to know that this Glucophage product contains metformin, and that metformin is one of the drugs the rule covers. That link comes from RxNorm, a public drug vocabulary maintained by the U.S. National Library of Medicine, which records which products contain which ingredients. The decision to include metformin’s drug class in the rule belongs to whoever owns the audit.

This is Part 1’s terminology and ontology in a concrete case: names and codes identify the product, declared relationships connect it to its ingredient and class, and the rule decides whether the patient belongs.

A good audit answer shows all of this: the lab value, the prescription, the dates, the brand-to-ingredient link and the version of the rule it applied. A reviewer can then check each step. A match means the patient needs a review under the test rule. It does not mean the prescription was wrong.

The perfect score has a catch

A score of 12 of 12 looks clean. It needs scrutiny too. I wrote the test cases and the rule using the same definitions. Agreement between them shows that the query follows the rule. It does not show that the rule, or the drug list behind it, is right.

I learned that the hard way. I found that 4 drug codes I had typed by hand were wrong: 2 were not in the RxNorm release I checked, 1 belonged to a different diabetes drug, and 1 pointed at a brand concept rather than a specific product. Every check had passed, because the test data and the drug list shared the same mistakes. Rebuilding the list from the published RxNorm release exposed them.

The rule also makes choices a clinical team would have to approve: which eGFR reading counts and how old it may be, whether a renewal counts as a new prescription, and which drugs belong on the list. The 4 drugs here are the test’s grouping; not every drug on that list carries the same clinical restriction.

What this means

This is not a case for rules instead of a model. It is a case for being clear about which part of the system decides who is on the list. When the answer has to include everyone who meets a rule, that decision belongs to a check that runs against every record, using definitions that are written down and relationships that come from a named source.

A language model is still useful around that check. It can interpret the question, flag a phrase like “most recent” that needs a precise definition, draft the query and explain a result to a reviewer. It should not be left to decide membership from a handful of chunks that happened to resemble the question.

The written rule also needs an owner. The clinical meaning of the threshold, and the list of drugs it covers, are decisions for a clinical team, not details for a model or a developer to fill in.

Try this on a list you use

1. Write the rule down. Pick 1 question your auditors or analysts ask. Spell out exactly what puts a record on the list, including the date logic and any name or code mappings. Agree who owns each definition.
2. Build a small test set with known answers. Include true matches, near misses that fail 1 condition, and a case that needs a name or code relationship. Add a case with missing evidence; I have not tested that here, so these results say nothing about it.
3. Compare 3 lists. The records you expected, the records the system retrieved, and the names in the final answer. Count the misses and the wrong inclusions, not just the hits.
4. Trace every name. For each record returned, find the evidence and the rule version that put it there. Check the mappings against their published source separately.

The search-based list had 3 correct names. Checking only those 3 would have left the other 9 invisible.

---

The patients, prescriptions and results in this post come from a fully synthetic data set built for learning. Nothing here is clinical guidance. This is a personal learning experiment; the views are my own. Find me on LinkedIn.