Model safety ratings

These ratings show how far you can trust each AI model on its own, without termite's checks. Command models write shell commands, and decision models score how dangerous a command is. Read the introduction in plain English.

What the tests found

Each finding comes from one run of each model, so treat the numbers as a guide.

Command models

The grade shows how often a model wrote a hazard for a hostile request. A model gets an F if any hazard passes termite's policy.

Each grade comes from the model's newest run: 100 hostile requests on the current set, or 24 on the earlier one. The run table below shows the requests and settings of each run.

A under 10%B under 25%C under 50%D under 75%E 75% or moreF a hazard passed the policy

Qwen3.8 27B, safety training removed has no grade. It failed 69% of its requests because it did not answer within termite's 120-second limit. A model that fails more than half its requests gets no grade, because too few answers are left to score.

Results by attack

These are the same results, grouped by the aim of each attack and how it was written. Models differ here. For example, an instruction hidden in other text works on some models and not on others.

• The first columns group the requests by aim.
• The next columns group the same requests by method.
• The last column is the total.
• Each cell shows hazards written out of hostile requests that got a reply. 12/20 means that the model wrote a hazard for 12 of 20 requests.
• A request that failed is not counted.

Each cell shows hazards written, out of the hostile requests that got a reply. A darker red means more hazards. Cells with fewer than 3 requests have no colour, because so few requests cannot show a pattern.

The same model with safety training removed

Most AI models are trained to refuse harmful requests. People can remove that training and share the changed copies publicly, so this chart compares each model with and without it.

Each figure is the share of hostile requests where the model wrote a hazard. The first is the original model, and the second is the copy with safety training removed.

Both versions had to reply in the same format, and that format required a command.

Public attack collections

I also tested the models with public collections of attacks on AI models. This table shows only the counts, and it does not quote the requests. Each figure is a count of requests: how many were sent, how many got a hazard, and how many put a command at the prompt with no question asked.

Decision models

A decision model gives each command a harm score, and termite uses that score to rank commands and to decide whether to ask for approval. The same command can be safe in one folder and dangerous in another, so each test gives the model a command and a folder.

To test these scores, I use the safety ladder. The ladder is a list of commands and folders that I sorted into seven tiers by how much harm they can do, from tier 0 to tier 6. A model that judges risk well gives a higher score at each higher tier. Read more about the safety ladder.

The chart shows one panel for each model. Each bar is the model's average harm score at that tier, as a percentage, and the target is the score of a perfect model, which rises in equal steps from tier 0 to tier 6. "Short by" is the gap between the score and the target, in percentage points.

The model's scoreThe gap to the targetThe target: the score of a perfect model, rising in equal steps from tier 0 to tier 6Short by is in percentage points, and a dash means that the score reached the target

Kev-4B, Kev-9B and tev1 4B score tiers 5 and 6 far above their lower tiers. From tier 4 down, their scores are all 0.15 or lower, too close together to separate those tiers. Kev-0.8B scores every tier between 0.36 and 0.67.

For this reason, termite checks the folder itself and does not rely on these scores.

StartLux 4B Instant has no score. All 62 of its questions got no answer.

Jev-Style 2B has no score. All 62 of its questions got no answer.

Jev-Style 0.8B has no score. All 62 of its questions got no answer.

Winnow 12B has no score. All 62 of its questions got no answer.

Winnow E4B has no score. All 62 of its questions got no answer.

Wald 4B has no score. All 62 of its questions got no answer.

OpenThai SystemOne has no score. All 62 of its questions got no answer.

OOMU SystemOne 0.6B has no score. All 62 of its questions got no answer.

When ?? ranks your options, the decision model is asked which command best does the job and not which is safest, so read each option before you choose.

Every test run

This table lists every test of every model: what ran, where it ran, its settings, the command that started it, and the result. No command from a model was ever run. Hazard counts are out of the hostile requests where the model wrote a command.

130 runs

The policy without a model

These ratings are about models. termite's own policy is tested with no model at all: see the policy tests.