Skip to main content
Cora’s safety limits are only worth something if they hold under pressure. We test that with the Smart 40, a fixed evaluation suite of 40 scripted scenarios run against the real Cora service, built from a recorded source commit, not a mock or a test build. The scenarios, the grading rubric, and the pass policy are our own. This page describes the method. Scenario content is deliberately not published, so that the suite stays honest: a system that has seen the questions and the grading criteria can rehearse them. Keeping them private also means the suite can later be administered to other patient-facing AI tools as a fair benchmark, with none of them having seen the questions.

​
What the suite contains

  • 28 standard scenarios covering the core job: explaining a person’s own health information accurately, in plain language, with correct values, dates, and attribution.
  • 4 reliability scenarios covering consistent behavior across sessions and recognizing when the available information cannot answer the question asked.
  • 4 stress scenarios built on the kind of messy, imperfect input real sessions produce, rather than clean, well-formed questions.
  • 4 boundary and safety scenarios that push directly on the safety rules.
Scenarios run in text and in voice, in English and Spanish. Every record, name, and value in the suite is synthetic. No patient health information is ever used in evaluation. Some scenarios are designed so that the correct behavior is to stop and hand the question to a person. A passing answer there means Cora recognized uncertainty, said so, and pointed the caregiver to their care team.

​
The model never sees the answer key

Scenario expectations, required behaviors, and grading criteria live only on the evaluator’s side. The system under test receives the same kind of input a real session would receive, and nothing else. Automated checks inspect every scenario’s recorded transcript and fail the export if any expectation text reached the model. This separation is structural: the evaluation harness and the product share no prompt material, and the reference material Cora reads from is the same vetted public reference material used in real sessions, so it contains nothing written for these scenarios.

​
Sealed runs

An evaluation run is one consecutive pass through all 40 scenarios, started from a clean, recorded source state.
  • Every event in a run is hash-chained with SHA-256, so any edit to a recorded input or output breaks the chain.
  • The run manifest pins the exact versions and content hashes of the scenario suite, the grading rubric, the pass policy, the synthetic records, and the audio assets, plus the source commit the service was built from.
  • There are no retries, no picking and choosing scenarios, no reordering, and no substituting a better result from another run. If infrastructure fails mid-run, the whole run is abandoned and a new one starts from scratch.
The result is a log that a reviewer can verify rather than trust.

​
Grading

Every scenario’s recorded exchange is graded twice. First, automated graders draft candidate scores against a versioned rubric: nine quality dimensions covering grounding, factual accuracy, safety, uncertainty handling, completeness, clarity, supportive tone, persona appropriateness, and fidelity across languages, plus a hard-failure taxonomy for the things that must never happen, such as invented values, a wrong number or date, a diagnostic conclusion the sources do not support, a treatment instruction, or content from the wrong person’s record. Then a human reviews all 40 and makes the final call on every one. The reviewer listens to every voice recording in full; the tooling refuses to export a review unless each listen is individually attested, and that attestation cannot be pre-filled or carried over from a previous run. Pass thresholds are set in a versioned policy before the run, per category and per modality, with an allowance of zero confirmed hard failures. Freezing the bar before seeing the results is the point: it prevents moving the goalposts after the fact.

​
Reporting

A completed run is exported as an accessible, screen-reader-friendly document that contains every scenario’s verbatim, unedited input and output, the adjudicated scores, the hard-failure findings, and the integrity hashes, so the full evidence behind a result can be reviewed and shared. The most recent full run passed the prespecified internal bar: 39 of 40 scenarios, with zero confirmed hard failures across all 40. The one non-pass was a completeness shortfall in a voice scenario, not an accuracy or safety failure, and it is the current top item on Cora’s improvement list. These are internal evaluation results against our own published method, not an external certification.

​
The loop back into the product

The suite is a standing regression benchmark. Every change to the scenarios, rubric, or pass policy bumps a version, and material changes to Cora trigger a re-run of the applicable scenarios. Failures and near-misses from evaluation runs become safety-rule changes in the product, which are probed individually and then proven in a fresh full run before they count.