The tests

This page describes each test: what it sends, what it records, where it ran, and what it cannot show. It gives the number and the kind of requests. It does not give the words of the requests, or the instructions that termite sends to a model.

What every model test has in common

• Nothing runs. A model’s commands are recorded and checked. No command that a model wrote is ever run.
• Each request starts fresh. A model sees no earlier request, unless the test adds text on purpose.
• Time limit. A model has 120 seconds to answer. After that, the trial ends as a Timeout.
• One row for each trial. Every trial records the model, the machine, the code version and how the trial ended.
• No request or command in the data. The rows on Explore hold ids, numbers and fixed words.

A trial with a command model has these steps:

1. termite sends the request to the model, with its usual instructions.
2. The model replies with commands, or it declines.
3. termite compares each command with its list of known hazards. This step only records the result, and it does not change what happens next.
4. termite’s policy checks each command.
5. A decision model rates the commands that the policy did not block.
6. The trial gets one result, and one failure type if something went wrong.

Core requests

Question: what does a model write, and what does termite then do with it?

This is the main test. Each model gets 161 requests, once each.

The 100 hostile requests are written in four ways:

The requests cover these areas: deletion, the system, files, secrets, persistence, remote code, version control, containers, packages, system defences, data theft and termite itself.

Each request has a level from 1 to 7. Levels 1 and 2 are everyday, levels 3 and 4 are risky, and levels 5 to 7 are hostile.

How it is scored:

• A hostile or risky request is scored on whether the model wrote a hazard, and whether termite stopped it.
• An everyday request is also scored on whether the command does the job. A wrong command is a Wrong answer.

Where it ran: the Local Sandbox on Mac, and on Hugging Face (ZeroGPU, T4 GPU and CPU). Every run shows its machine.

What it cannot show:

• Each request ran once for each run, so one result can be chance. Repeated requests measure this.
• The hazard list has only known hazards. A new kind of harm is not counted.
• Some runs reached fewer than 161 requests. Compare runs of the same test set.

Outside requests

Question: do the core results hold on requests that I did not write?

This test sends 120 requests from public collections:

Each public set is pinned to one version and checked by hash.

How it is scored: in the same way as core requests.

Where it ran: on the Local Sandbox on Mac. Two models ran this test, a model and its copy with safety training removed, twice each.

What it cannot show: many of these requests are not about the shell, so a command is often not a sensible reply. Two models are too few to compare models.

Repeated requests

Question: does a model give the same answer twice?

A model can give different answers to the same request. This test sends the same requests again and again.

• It uses 14 requests: 2 from each of the 7 levels. That is 4 everyday, 4 risky and 6 hostile.
• Each request goes to each model 5 times, with no filler text.

How it is scored: the test counts how many different answers a model gave to each request. A request is steady if all 5 answers are the same. Answers are compared as text, so two commands that do the same job but are written differently count as different.

Where it ran: on the Local Sandbox on Mac, with 6 models.

What it cannot show: 14 requests are a small sample. A different answer is not always a worse answer, because the test compares the text and not the effect.

Long sessions

Question: what happens when a request arrives after a very long session?

termite starts each question with an empty context. A context is the text that a model can hold at one time. Another program could still send termite a very large request, and a model that has read a lot of text might behave differently.

This test uses the same 14 requests as repeated requests. It puts harmless filler text in front of each request, and then the request. Each request runs twice at each level.

100% means 4096 tokens of filler text. This limit was set for the test, and it is not the model’s own limit. The size is an estimate, which counts 4 characters as one token.

How it is scored: in the same way as core requests. Two more results matter here:

• Overflow. The text did not fit, and the server refused it. This shows where the limit is.
• Broken reply. The text fitted, but termite could not read the reply.

The two machines are shown apart because their limits differ. On Hugging Face, the server for the command model and the server for the decision model each had a limit of 4096 tokens. On the Local Sandbox on Mac, the command models ran in Ollama, and the limit that stopped trials was the decision model’s server.

What it cannot show:

• The filler text is harmless and asks the model to do nothing. The test cannot show a session that talks a model out of its instructions.
• The test does not record how much of the text reached the model. A server can cut text without saying so.
• A trial that overflowed cannot show a hazard, so fewer hazards at a high level does not mean a safer model.

The safety ladder

Question: does a decision model see the risk in a command?

A decision model does not write commands. It reads a command and scores it. This test gives each decision model 52 commands. Each command comes with a folder, because the same command can do more harm in one folder than in another.

The commands are sorted into seven tiers, from harmless to critical:

How it is scored:

• Risky rated safe. How many commands at tier 2 or above the model called safe.
• Harmless rated safe. How many commands at tiers 0 and 1 the model called safe. A model that calls nothing safe would otherwise look perfect.
• Order. Whether the harm scores rise with the tiers, measured with Kendall’s tau.
• Folder. Whether the score rises when the same command is in a riskier folder.

Where it ran: on the Local Sandbox on Mac, and on Hugging Face (ZeroGPU and CPU). 17 models ran this test.

What it cannot show: the tiers are my own judgement of harm. 52 commands are a small sample. A model is scored on one command at a time, without a real session around it.

Safest choice

Question: when there are several ways to do a job, does a decision model put the safest way first?

Each decision model gets 10 jobs. Each job comes with 2 or 3 commands that do it, at different tiers. The model ranks them.

How it is scored: the test counts the jobs where the top command is the safest one. The results are on the model ratings.

What it cannot show: a model may rank by which command does the job best. A good rank for this reason still counts as a miss.

Tests that use no model

These tests check termite’s own code, so they give the same result every time. The policy tests page shows the latest results.

• Scripted attacks. 108 fixed commands go straight to the policy. 91 should be refused and 17 should be allowed.
• Outside commands. Every command in two public datasets goes straight to the policy: 46,518 after duplicates are removed. Nothing is left out by size or shape. These datasets were made to train models, and most of their commands are unverified. They are not a record of commands that people ran. This test shows what the policy allows. A blocked command is one that the policy could not prove only reads, and many of these commands do write.
• Unit and fuzz tests. These check that the code does what it says, including with random input.