Same Question, Different Answers: Governing an AI Data Analyst

David Do, MD^1,2 · Marc Tobias, MD²

¹Adjunct Associate Professor of Neurology, University of Pennsylvania ²Phrase Health, Inc.

Summary

An agent based on Claude Code and Claude Sonnet 5 was able to write SQL queries and generate reports on a health database with only prompting and no additional training. The agent was told nothing specific to healthcare. Without local definitions, its decisions varied from run to run. When given data definitions in a knowledge library, the agent adhered to it and returned precise, reproducible answers.

Introduction

AI analytics tools promise to make analytic work faster and enable anyone to pull data without knowing how to code. This adds the risk of reporting without context and with divergent definitions.

At a health system, "heart failure cohort" might mean a specific set of diagnosis codes, drawn from visit diagnoses but not the problem list. A data analyst draws on experience to apply "local" definitions like these, some documented and some remembered, making many decisions in every query. Even trained analysts are known to vary in their choices, leading to widely varying results (Silberzahn et al.).

How can we use AI to let anyone generate reports, while ensuring quality of reporting and adherence to an organization's own definitions?

Methods

An AI report-writing agent was built with the Claude Agent SDK, using Claude Sonnet 5 as the model, run in a container, with access to a set of tools and a knowledge library (Figure 1). The 2,248-word system prompt instructs on desired report structure, tables and charts, footnotes, appendix, disclosure of defensible alternatives, and how to record, read back, and publish analyses. A 4,572-word Database Overview document describes available tables. A user chats with the agent and can run multiple analyses, view results, and publish a report.

Figure 1: Agent architecture

User asks a question in a web browser

Knowledge library

Items are tagged so that those relevant to a question are appended to it.

Claude agent

System prompt

Role, analytic conventions, reporting expectations. No instructions are specific to healthcare.

File reading functions

ReadGlobGrep

Database functions

run_sqllist_tablesdescribe_table

Report writing functions

record_analysispublish_report

Database overview

Relations and columns · join cardinalities · freshness

Data warehouse

Live EHR-derived tables, queried read-only via run_sql

Storage

A cloud server stores the generated reports

Experiment

One question was paraphrased ten ways and run in four arms (Figure 2): without a knowledge library, and with libraries of 8, 100, and 1,000 items. For each of eight decision forks (Table 1), one option was randomly chosen to serve as the "institutional definition"; the definitions (1,347 characters, ~336 tokens) were embedded in the knowledge libraries. Outcomes: (1) primary, the standard deviation of the reported proportion; (2) the option chosen at each fork; (3) the share of variance attributable to phrasing, by one-way random-effects ANOVA with phrasing as the grouping factor (k=10, n=98); (4) concordance, the percentage of fork decisions matching the library's definition, by library size.

Figure 2: Experimental design

One general question

"Among patients seen in cardiology clinic with heart failure in 2024, what proportion were prescribed a beta blocker?"

Paraphrased ten ways

No definitions given to agent

n=100 trials

8 definitions in an 8-item library

n=100

8 definitions in a 100-item library

n=50

8 definitions in a 1,000-item library

n=50

Table 1: Key decision forks analyzed

Fork

Decision

Possible pathways

Patient-diagnosis linkage

Problem list · visit diagnosis · hospital billing code · any of the three

Heart-failure code set

I50 prefix · enumerated subtype list · a value set · text match on "heart failure"

Beta-blocker vocabulary

Pharmacologic class name · pharmacologic class id · therapeutic class · a hand-listed set of drug names

"Cardiology clinic"

Dept. specialty · provider specialty · order dept. · name match

Active departments

Active departments only · all departments ever

Encounter type

Office visits only · all ambulatory contacts · every encounter type

Order status

Signed and active · any order placed · dispensed or administered

Deceased patients

Include deceased patients · exclude them · exclude only if died before year end

Figure 3: Example of an item in the knowledge library

SAMPLE INSTITUTIONAL DEFINITION

"Cardiology clinic" means the department the visit took place in has specialty Cardiology. Not the provider's specialty, not the department attached to an order, and not a department-name pattern match.

Results

How much does the final answer vary?

18.3 → 4.2

SD of the reported proportion, in points, without vs with an 8-item knowledge library

Figure 4: Reported proportion prescribed a beta blocker; each dot is one run

Response

0%

20%

40%

60%

80%

median
64.4%

median
60.1%

No definitions

n=98 · SD 18.3 pts · range 8.7–76.5%

8-item library

n=98 · SD 4.2 pts · range 42.1–73.9%

Of 100 runs, two in each arm reported a patient count rather than a proportion and are not plotted.

How much variation is attributable to differences in phrasing?

10.5%

of variance came from phrasing in the no-definitions arm (ICC 0.105, 95% CI 0.000–0.192). Most came from the agent itself.

What decisions were made at each fork?

Figure 5: Decisions made at each fork (when no definitions are given)

Visit diagnosis 45

Problem list 41

I50 prefix 88

Pharmacologic class (id) 42

Pharmacologic class (name) 36

Department specialty 60

Order department 21

All departments ever 100

Every encounter type 100

Any order placed 92

Include deceased patients 100

Each bar is one fork, split by the options the runs actually took, commonest first. Counts are runs (n=100).

Do eight definitions get diluted in a larger knowledge library?

93% → 89%

concordance as the library grew from 8 to 1,000 items: a modest decline, with overlapping confidence intervals.

Figure 6: Decisions concordant with definitions, by library size

Share of fork decisions concordant with the library

0%

25%

50%

75%

100%

93.1%

91.7–94.6

n=100

92.0%

89.2–94.8

n=50

89.0%

86.5–91.5

n=50

8-item library

100-item library

1,000-item library

Items in the knowledge library

Conclusions

Like the 29 analyst teams in Silberzahn et al., an AI agent given the same question makes defensible but different choices. Unlike human teams, it can be handed the institution's definitions every time, and it follows them: the SD of the reported proportion dropped from 18.3 to 4.2 points. Concordance held near 90% even when the eight rules sat among 1,000 items.

A knowledge library turns local knowledge from something each analyst carries in their head into something every query receives.

Limitations:This work focused on the precision of the answer, not accuracy. Future work will focus on query accuracy versus a human-defined gold standard. How performance scales with larger table schemas is critical to address next.

References

Silberzahn R, Uhlmann EL, Martin DP, et al. Many analysts, one data set: making transparent how variations in analytic choices affect results. Adv Methods Pract Psychol Sci. 2018;1(3):337-356.

Contact

David Do, MD david.do@phrasehealth.com
Marc Tobias, MD marc@phrasehealth.com
phrasehealth.com

Same question, different answers: governing an AI data analyst with local definitions

Introduction

Methods

Experiment

Results

How much does the final answer vary?

How much variation is attributable to differences in phrasing?

What decisions were made at each fork?

Do eight definitions get diluted in a larger knowledge library?

Conclusions

References

Contact

Get started with your personalized demo