SIM-VAIL
Whether a model responds safely to simulated vulnerable users across adaptive multi-turn conversations.
- Grading
- LLM judge rates each full conversation on 39 dimensions (1–10)
What this is
The AI Safety Index gives each AI model configuration one score for how safely it behaves across every area we test, from emotional support and health advice to privacy and reliability.
It combines every benchmark we publish. Each counts according to a provisional weight, so a model can be compared at a glance, with each benchmark’s own result a click away.
Index composition
The overall index is a weighted average of the benchmarks below. Each safety area’s share is the sum of its benchmarks’ weights.
| Safety area / Benchmark | Weight | Items | Repeats | Format | Grader |
|---|---|---|---|---|---|
| Mental & Emotional Safety | 20% | ||||
| SIM-VAIL | 10% | 90 conversations (30 profiles) | 3 | Multi-turn, simulated user | Claude Opus 4.5 judge; Claude Sonnet 4.5 auditor |
| Spiral-Bench | 10% | 30 conversations of 20 turns | 1 | Multi-turn, simulated user | GPT-5 judge; GPT-4.1 simulated user |
| Youth Safety | 12% | ||||
| KORA | 12% | 737 scenarios | 1 | Multi-turn (3 turns), simulated child | GPT-5.2 judge (high); GPT-4.1 simulated child |
| Medical Advice Safety | 15% | ||||
| PatientSafetyBench | 7% | 466 queries; headline on 99 | 1 | Single-turn | o3-mini judge (medium) |
| HealthBench-Hard | 8% | 1,000 conversations | 1 | Single reply to a conversation | GPT-4.1 grader |
| Manipulation | 21% | ||||
| ELEPHANT | 8% | 3,027 open-ended advice queries | 1 | Single-turn | GPT-4o judge |
| DarkBench | 8% | 660 prompts | 1 | Single-turn | GPT-4o judge |
| HumanAgencyBench (Autonomy) | 5% | 1,000 prompts (500 per dimension) | 1 | Single-turn | o3 judge |
| Security | 6% | ||||
| ASK — AI Scam Knowledge | 6% | 420 prompts (35 scenarios × 4 cue levels) | 3 | Single-turn | Claude Opus 4.8 judge |
| Bias & Fairness | 10% | ||||
| FairMT-Bench | 10% | 990 five-turn dialogues | 1 | Multi-turn, scripted | GPT-4 Turbo judge |
| Privacy / Confidentiality | 3% | ||||
| ConfAIde | 3% | 270 scenarios in the headline tier | 10 | Single-turn | Rule-based, no judge |
| Misinformation | 7% | ||||
| SimpleQA Verified | 3% | 1,000 questions | 1 | Single-turn, no tools | GPT-4.1 grader |
| SYCON-Bench | 2% | 100 debate dialogues | 1 | Multi-turn, scripted (5 turns) | GPT-4o judge |
| HumanAgencyBench (Correct Misinformation) | 2% | 500 prompts | 1 | Single-turn | o3 judge |
| Rule Following | 6% | ||||
| SystemCheck / RealGuardrails | 3% | 239 handwritten cases | 1 | Single reply to a conversation | GPT-4o judge |
| AgentIF | 3% | 707 instructions (8,440 constraints) | 1 | Single reply to a conversation | GPT-4o judge |
| Total | 100% | ||||
The weights are value judgments on harm severity, how directly each metric measures harm, and overlap between benchmarks. They are not fitted estimates, and average scores do not set them.
What each benchmark measures
Spiral-Bench
Whether a model pushes back on, or reinforces, delusional, manic or conspiratorial spirals over long chats.
- Grading
- LLM judge scores each turn, then the released Safety Score composite
KORA
Whether a model keeps children aged 7–17 safe across 25 risks in short conversations with a simulated child.
- Grading
- LLM judge grades each transcript failing / adequate / exemplary (0 / 0.5 / 1)
PatientSafetyBench
How harmful a model's answers are to risky medical questions asked from a patient's perspective.
- Grading
- LLM judge rates harmfulness 1–5
HealthBench-Hard
How well a model's reply in hard, realistic health conversations meets physician-written rubric criteria.
- Grading
- LLM grader checks each rubric criterion as met or not
ELEPHANT
How often a model is socially sycophantic in advice: excessive validation, indirectness and accepting the user's framing.
- Grading
- LLM judge labels each behaviour present or absent
DarkBench
How often a chatbot's reply shows one of six manipulative dark patterns, such as brand bias or user retention.
- Grading
- LLM overseer judges whether the targeted pattern is present
HumanAgencyBench (Autonomy)
Whether a model leaves a person's values and important decisions to them rather than steering them (HumanAgencyBench's Avoid Value Manipulation and Defer Important Decisions dimensions).
- Grading
- LLM judge lists rubric issues; each prompt scores 10 minus deductions
ASK — AI Scam Knowledge
Whether a model spots a scam the user describes and warns them, as scam cues increase.
- Grading
- LLM judge scores each reply 0–4 on a released rubric
FairMT-Bench
Whether a model produces biased content by the fifth turn of dialogues designed to draw out social bias.
- Grading
- LLM judge decides whether the fifth reply is biased
ConfAIde
Whether a model reveals private information where social norms say it should stay confidential.
- Grading
- String matching against who should and should not learn the secret
SimpleQA Verified
How accurately a model answers short factual questions from its own knowledge, and whether it abstains instead of guessing.
- Grading
- LLM grader labels correct / incorrect / not attempted
SYCON-Bench
How many turns a model holds its position under repeated user pushback before giving in.
- Grading
- LLM judge checks each turn against the original stance
HumanAgencyBench (Correct Misinformation)
Whether a model notices and corrects a false claim built into a user's request (HumanAgencyBench's Correct Misinformation dimension).
- Grading
- LLM judge lists rubric issues; each prompt scores 10 minus deductions
SystemCheck / RealGuardrails
Whether a model obeys the guardrails in a real-world system prompt when users push against them.
- Grading
- LLM judge decides whether the system instructions were followed
AgentIF
Whether a model satisfies every constraint in long, realistic instructions from agent applications.
- Grading
- Per-constraint checks by LLM judge and code
How the index is calculated
index = Σ (weight × published score) ÷ Σ (weights of the benchmarks with a result)- Every input is a benchmark’s published 0–100 score, used as it is. No score is normalized, before or after averaging.
- A missing result is left out, never counted as zero; the remaining weights are rescaled for that model.
- The index summarizes evaluations. It is not a percentage of safety or a guarantee about any product.
Testing parameters
These conditions apply to every benchmark in every index.
- Published scores
- Each benchmark contributes one headline metric, converted to a 0–100 scale where higher is better. Values past either end are clipped.
- Model configurations
- The reasoning setting is part of each configuration. Configurations are published separately and never averaged together.
- Sampling
- Benchmarks set no temperature for the tested model; each provider's default applies.
- Output limit
- The lower of the model's own maximum and the benchmark's cap, which is 65,536 tokens where one is set.
- Retries
- Failed API calls are retried up to 3 times with backoff; a stopped run is resumed up to twice.
- Provider refusals
- An explicit provider safety block is recorded, not retried. DarkBench, FairMT-Bench, ELEPHANT and KORA score it as the safe outcome; elsewhere it is left unscored.
- Judge failures
- A refused or failed judge, simulator or auditor call is an execution failure and is never scored.
- Which run is published
- The newest complete full run on the pinned protocol. Partial runs never replace complete ones.
- Scope
- Scores describe API model configurations. Consumer apps may add their own prompts, tools and safeguards.
What this does not measure
The index covers the behaviours its benchmarks test, almost entirely in English text conversations. It does not measure images, voice, or actions an assistant takes in the world; how capable or useful a model is; or how a particular app wraps it.
A high score is not a certification that a model is safe, and a low one does not mean every conversation will go badly.
Using this data
Cite the index by name with the date of the published data shown on each chart, and read the benchmark pages before drawing conclusions from a difference of a few points.
The complete published score table is available as JSON at /api/scores.