Blog
AI in medical education Dr. Alastair Dunne

How should medical schools evaluate AI practice tools?

The evidence for AI practice tools is early, and trials in medical education are hard to run. A tool can still be judged in stages, starting with safety.

A clipboard with an assessment checklist.

The first thing most educators ask of an artificial intelligence (AI) practice tool is whether it makes learners better. The evidence on that is still thin, and randomised trials in medical education are hard to run well.

Waiting for those trials has a cost, because each cohort passes through training only once.

So judge a tool in stages. Start with safety: the tool shouldn't mislead learners or leave them worse off than not practising at all. That can be checked now, from how the tool is built. Whether learners use it, and whether they get better, can be measured as use grows.

What evidence is there for AI practice tools?

A Canadian scoping review of 21 studies of large language model (LLM) virtual patients found that most were pilot or feasibility studies, with a median sample of 21. Assessment was mostly self-report, and no study measured changes in behaviour or effects on patients (Gandhi et al., 2026, AI in Medicine).

Why randomised trials in medical education are hard

Randomised trials are harder to run in education than in clinical medicine. An editorial in the Journal of Graduate Medical Education set out why. Learners often can't be blinded to the group they're in. They share what they learn, which contaminates the comparison. Numbers are often small. And teaching that is critical to learning can't be withheld from a control group (Sullivan, 2011, Journal of Graduate Medical Education).

Even a well-run trial struggles to isolate one part of a programme. Learners pick up the same skills from lectures, placements, question banks and each other. Geoff Norman of McMaster University argued that the effect of a single educational intervention is likely to be lost in that variation (Norman, 2003, Medical Education). Studies comparing a new technology with an older method are especially hard to read, because too many things differ at once (Cook, 2012, Advances in Health Sciences Education). An AI tool adds one more variable: the model underneath can be updated part-way through a study.

Liking a tool and feeling more confident still count

Whether learners like a tool still matters. A tool they don't use does nothing for them, however well it's built.

Confidence counts too, if it's measured well. Self-efficacy is a learner's belief that they can do a specific task. In a study of second-year medical students, 71% of those with high self-efficacy scored above the mean on their objective structured clinical examination (OSCE), against 51% of those with low self-efficacy (Mavis, 2001, Advances in Health Sciences Education). A well-built confidence measure is useful early evidence while performance data build up.

Do we need a trial to know practice with feedback works?

Some things in medicine are done without a randomised trial because the mechanism isn't in doubt. A BMJ paper made the point by systematically reviewing randomised trials of parachutes and finding none (Smith and Pell, 2003, BMJ). That only works where the mechanism is well established.

Repeated practice with feedback is that kind of mechanism. A meta-analysis of 14 studies found that simulation with deliberate practice was better than traditional clinical education for acquiring specific clinical skills, with an overall effect size of 0.71 (McGaghie et al., 2011, Academic Medicine). Cook has argued that research still asking whether learners can learn from this kind of teaching is asking a settled question. We set out how medicine builds consultation skill in deliberate practice in medical education.

For an AI practice tool, the open question is narrower: whether its practice and feedback are sound enough not to mislead learners.

The first test: does it leave learners worse off?

A tool passes the safety test when it won't teach wrong medicine or bad habits. That depends on how it's built, and an institution can check it before any outcome data exist:

  • The clinical content comes from structured knowledge that clinicians can review, rather than from a general model improvising each time. Our guide to what to look for in an AI OSCE platform covers how to check.
  • The tool is fitted to the curriculum and the learner's stage, so it practises what the programme teaches.
  • Feedback is formative and specific, and the institution can see how its cohort is using the tool.
  • A person makes every decision about whether a learner passes or progresses, as we set out in who decides where AI belongs in assessment.

A general chatbot playing a patient is unlikely to meet these criteria, so it is unlikely to be safe for institutional use.

Evidence in stages

Clinical AI already works this way. The DECIDE-AI reporting guideline describes an early stage of evaluation, at small scale, to check an AI system's safety and the human factors around its use before larger trials (Vasey et al., 2022, Nature Medicine). It was written for clinical decision support, not education, but the logic carries over.

StageQuestionEvidence that answers it
Before adoptionCould it mislead learners or leave them worse off?How it's built: structured clinical knowledge, fit to the curriculum, formative feedback, a person making every summative decision
Early useDo learners use it and value it?Uptake, satisfaction and task-specific self-efficacy
As use growsAre learners getting better?Objective performance measures read alongside self-report, and comparisons with other ways of practising

Waiting has a cost

Supervised practice hasn't grown with student numbers, as we set out in why the world is training more doctors. A programme that waits for a definitive trial before offering more practice gives its current learners less of it.

Where Gestalt fits

Gestalt builds consultation practice with formative feedback, grounded in the Gestalt knowledge graph, a structured body of clinical knowledge. An institution should ask us for the evidence at each stage, as it would any vendor.