# Decode a data format

> **Skill `re-format`.** Decode a game's data files - package/archive containers, compressed entries, resource and asset formats, save games - by finding the code that reads them, prototyping a decoder with tools/formats/reader.py, proving it against every file of its type with validate.py (all bytes consumed), then porting it. Use when a file type is not yet understood or a decoder fails on some files.

A decoder is right only when it reads **every file** of its type to the **last byte**. One
sample proves little: optional fields, versions and rare cases show up in a few files only.

## 1. Look at the bytes

```bash
xxd target/data/units.dat | head -20
```

Look for: a magic number at the start; counts and sizes (small integers that match the file's
length or the number of something visible); offsets (values that point inside the file);
length-prefixed strings (a small integer followed by that many printable bytes); repeated
records of a fixed stride; float values (`0000803f` is 1.0, `0000003f` is 0.5). Measure entropy
(`re-inventory` section 4): near 8.0 means compressed or encrypted.

## 2. Find the reader in the code

Guessing a format from bytes alone is slow and fragile. Find the code that reads it:

- the file's name or extension in the strings (`decomp_query.py strings .dat`), then the
  function that opens it, then the function that parses the opened bytes;
- the magic number as an integer constant (`decomp_query.py find 0x4c504d53`);
- for engines with a serializer pattern (`Serialize(Stream&)`), the class's serializer: it reads
  each field in order, with its type, and its version checks (`if (version > 2)`).

Write down each read in order: offset, type, meaning, and the condition under which it is read.

### Measure the reads (DOS games)

Run the original under DOSBox-X with the file log, then map the reads per file:

```bash
scripts/dosbox-run.sh --fileio --time 30 target/game GAME.EXE
python3 tools/dos/fileio_trace.py analysis/dosbox/dosbox.log --game target/game --file LEVELS.DAT
```

A program that reads a file field by field shows its record layout in the read sizes:
`[2, 1, 4] x 120 (stride 7)` is 120 records of a word, a byte and a longint. A 1-byte read
followed by a read of varying length is a Pascal string. This is *measured* evidence; name the
run (program, inputs, time). With Borland debug symbols, `tools/dos/tdinfo.py BINARY types NAME`
gives the record type the reader fills, field by field.

## 3. Containers and compression

Most games pack files into an archive: a header, a table of entries (name or hash, offset, size,
compressed size, flags), then the data. Find the table reader first; then each entry is an
ordinary file. Common compression: zlib/deflate (`78 9c`), LZ4, zstd (`28 b5 2f fd`), LZO, LZF,
LZMA, Oodle (Kraken/Mermaid: a proprietary library, look for `oo2core` DLLs). If the bytes are
encrypted, the key and the algorithm are in the reader: trace them, do not brute-force.

Extract entries into `derived/extracted/` (ignored by git), never into tracked folders.

## 4. Prototype the decoder in Python

```python
# decoders/units.py
def decode(r):                       # r is a tools/formats/reader.py Reader
    r.magic(b'UNIT')
    version = r.u32()
    units = []
    for _ in range(r.u32()):
        u = {'id': r.u32(), 'name': r.pstr32(), 'speed': r.f32()}
        if version >= 2:
            u['armour'] = r.u16()
        units.append(u)
    return units
```

`reader.py` gives bounds-checked reads (`u8 u16 u32 i32 u64 f32 f64 bool bytes cstr pstr8
pstr32 wstr`), `magic()`, and `block(size)` for a length-prefixed section that must be consumed
exactly. Then run it over every file:

```bash
.venv/bin/python tools/formats/validate.py decoders/units.py 'derived/extracted/**/*.unit' --show 1
```

Every file must report complete. A failure says where it stopped and shows the next bytes: a
missed field, a wrong type or a version branch. `examples/sample-game/decode_sample.py` is a
complete small example.

## 5. Port and test

Port the decoder to the reimplementation's language as a plain reader with no dependencies, one
function per type, documenting each field's meaning and naming the original reader function.
Its test opens the real files from the player's game, **skips with a message when they are not
there** (they are never committed), parses every file and spot-checks a known value.

## 6. Save games

Saves are the strictest test of a format: they hold the whole state of a running game. Find the
save function (strings: the save folder, the extension), trace it like a serializer, and decode
real saves of different sizes and ages. Saves from an older game version may use older
branches: keep them in the corpus. Later, a reimplementation that writes saves must produce
files the original game loads.

## Pitfalls

- Files in one folder can be different types: select by magic or type id, not by folder.
- Two empty strings in a row look like a zero integer: confirm with a file that has text.
- Booleans are often one byte; enums four; colours four floats or one packed integer: the reader
  code says which.
- Alignment and padding: some formats align records to 4 or 16 bytes.
- Never commit sample game files as test fixtures; generate synthetic ones if a test needs them.