Decode a data format

Skill re-format. Decode a game's data files - package/archive containers, compressed entries, resource and asset formats, save games - by finding the code that reads them, prototyping a decoder with tools/formats/reader.py, proving it against every file of its type with validate.py (all bytes consumed), then porting it. Use when a file type is not yet understood or a decoder fails on some files.

Copy this prompt into your coding agent. Replace the part in angle brackets.

In my GameFrame project, use the re-format skill for <the task>.
Read the skill first and follow its steps in order.
Report the evidence it asks for, with its label: observed, measured, inferred or unknown.

A decoder is right only when it reads every file of its type to the last byte. One sample proves little: optional fields, versions and rare cases show up in a few files only.

1. Look at the bytes

xxd game/data/units.dat | head -20

Look for: a magic number at the start; counts and sizes (small integers that match the file's length or the number of something visible); offsets (values that point inside the file); length-prefixed strings (a small integer followed by that many printable bytes); repeated records of a fixed stride; float values (0000803f is 1.0, 0000003f is 0.5). Measure entropy (re-inventory section 4): near 8.0 means compressed or encrypted.

python3 tools/triage/filetypes.py game/data/units.dat --blocks 4096   # where the dense parts are
python3 tools/formats/xorscan.py game/data/script.dat                 # a simple XOR or add key?
python3 tools/formats/xorscan.py game/data.pak --records DIR:28:4:16 --key-in game/GAME.EXE   # names XORed in each directory record
python3 tools/formats/shadetable.py game/data/shade.rmp --offset 14 --levels 64 --palette PAL --png derived/shade.png   # a light or fog table
python3 tools/formats/timestamps.py --scan saves/slot1.sav --from 1996 --to 1998   # a saved date?
python3 tools/triage/similar.py saves/before.sav saves/after.sav                  # how much changed
python3 tools/triage/constants.py game/GAME.EXE        # the decompressor or checksum, by its constants
python3 tools/triage/binview.py game/data/units.dat derived/units.png --width 24   # records show as stripes
python3 tools/formats/rawimage.py game/data/TITLE.SCR derived/title.png --mode linear8   # a raw picture?
python3 tools/formats/markers.py plant saves/slot1.sav derived/marked.sav --from 0x40   # follow fields into memory
python3 tools/formats/fields.py game/data/units.dat --stride 24 --png derived/units.png   # what each offset holds
python3 tools/formats/skeleton.py game/models/hero.mdl    # parent arrays, quaternions, joint names
python3 tools/formats/shaders.py game/data/shaders.pak --dump derived/shaders   # compiled and source shaders

Text that turns readable under one XOR byte or a short key is a common way games hid their scripts and tables. xorscan.py tries those weak schemes and shows a preview of each. Confirm the scheme in the reader's code before you port it. An archive's directory often XORs only the names: zero padding after each name then stores the key itself, and --records reads those bytes and finds the whole key in the executable. A palette game draws light and fog through a table of levels: shadetable.py finds the level that leaves every colour unchanged, which is the game's normal light. Palettes of 6-bit values are often shifted left by 2 (63 is 252), not scaled: pass --dac shift to rawimage.py and catalog.py when the loader shifts. Two saves made a minute apart, with one setting changed, show which bytes hold that setting and where the save time is.

2. Find the reader in the code

Guessing a format from bytes alone is slow and fragile. Find the code that reads it:

• the file's name or extension in the strings (decomp_query.py strings .dat), then the function that opens it, then the function that parses the opened bytes;
• the magic number as an integer constant (decomp_query.py find 0x4c504d53);
• for engines with a serializer pattern (Serialize(Stream&)), the class's serializer: it reads each field in order, with its type, and its version checks (if (version > 2)).

Write down each read in order: offset, type, meaning, and the condition under which it is read.

Measure the reads (DOS games)

Run the original under DOSBox-X with the file log, then map the reads per file:

scripts/dosbox-run.sh --fileio --time 30 game GAME.EXE
python3 tools/dos/fileio_trace.py analysis/dosbox/dosbox.log --game game --file LEVELS.DAT

A program that reads a file field by field shows its record layout in the read sizes: [2, 1, 4] x 120 (stride 7) is 120 records of a word, a byte and a longint. A 1-byte read followed by a read of varying length is a Pascal string. This is measured evidence; name the run (program, inputs, time). With Borland debug symbols, tools/dos/tdinfo.py BINARY types NAME gives the record type the reader fills, field by field.

3. Containers and compression

Most games pack files into an archive: a header, a table of entries (name or hash, offset, size, compressed size, flags), then the data. Find the table reader first; then each entry is an ordinary file. Common compression: zlib/deflate (78 9c), LZ4, zstd (28 b5 2f fd), LZO, LZF, LZMA, Oodle (Kraken/Mermaid: a proprietary library, look for oo2core DLLs). If the bytes are encrypted, the key and the algorithm are in the reader: trace them, do not brute-force.

Extract entries into derived/extracted/ (ignored by git), never into tracked folders.

4. Prototype the decoder in Python

# decoders/units.py
def decode(r):                       # r is a tools/formats/reader.py Reader
    r.magic(b'UNIT')
    version = r.u32()
    units = []
    for _ in range(r.u32()):
        u = {'id': r.u32(), 'name': r.pstr32(), 'speed': r.f32()}
        if version >= 2:
            u['armour'] = r.u16()
        units.append(u)
    return units

reader.py gives bounds-checked reads (u8 u16 u32 i32 u64 f32 f64 bool bytes cstr pstr8 pstr32 wstr), big-endian reads (u16be i16be u32be i32be f32be) for Amiga, Mac and console data, variable-length numbers (uleb sleb), bit fields (bits(n), align()), magic(), and block(size) for a length-prefixed section that must be consumed exactly. Then run it over every file:

.venv/bin/python tools/formats/validate.py decoders/units.py 'derived/extracted/**/*.unit' --show 1

Every file must report complete. A failure says where it stopped and shows the next bytes: a missed field, a wrong type or a version branch. examples/sample-game/decode_sample.py is a complete small example.

For editors and mods after the writer, read docs/working/mods.md. Before writing or modding a file, add encode(result) to the decoder and run the same command with --write: every file must come back identical, byte for byte. A failure names the first offset that differs. Keep the stored form beside the friendly view (raw palettes, pools as stored, unused tails) until it passes (docs/knowhow/writer-round-trip-every-file.md).

Then look at what the decoder found (re-browse). Add images(result) to the decoder when the format holds pictures, and run the catalogue with it:

python3 tools/formats/catalog.py game derived/catalog --decoder '*.unit=decoders/units.py'

Open derived/catalog/index.html: a wrong width slants a picture, a wrong palette changes its colours, a misread type shows absurd values. When the port has a viewer, add a view for the format there too.

5. Port and test

Port the decoder to the reimplementation's language as a plain reader with no dependencies, one function per type, documenting each field's meaning and naming the original reader function. Its test opens the real files from the player's game, skips with a message when they are not there (they are never committed), parses every file and spot-checks a known value.

If the project writes the format too (an editor, a mod tool, saves), prove the writer with a round trip: read every file, write it back, and compare the bytes. Each difference is a field the writer loses.

6. Save games

Saves are the strictest test of a format: they hold the whole state of a running game. Find the save function (strings: the save folder, the extension), trace it like a serializer, and decode real saves of different sizes and ages. Saves from an older game version may use older branches: keep them in the corpus. Later, a reimplementation that writes saves must produce files the original game loads.

Pitfalls

• Files in one folder can be different types: select by magic or type id, not by folder.
• Two empty strings in a row look like a zero integer: confirm with a file that has text.
• Booleans are often one byte; enums four; colours four floats or one packed integer: the reader code says which.
• Alignment and padding: some formats align records to 4 or 16 bytes.
• Never commit sample game files as test fixtures; generate synthetic ones if a test needs them.

Agents and tools

Agents that preload this skill: re-decoder.

Tools this skill names:

• fileio_trace: Turn a DOSBox-X file log into a per-file map of reads: which file, at which offset, how many bytes, in what order.
• tdinfo: Read Borland Turbo Debugger symbols (TDINFO, signature FB52) from a DOS or NE executable.
• binview: Draw a file as a picture, one pixel per byte (or per block of bytes), to see its structure at a glance: headers, tables of records, text, code, compressed or encrypted parts, padding.
• constants: Name the algorithms in a binary by their constants: random number generators, checksums, hashes, ciphers, compressors and well-known maths tricks.
• decomp_query: Answer questions about a Ghidra export (scripts/ghidra-export.sh) without reading it whole.
• filetypes: What each file of a game is, from its bytes: the format its first bytes name, its size and its entropy.
• similar: How alike two files are, when they are not identical: two versions of an executable, a patched data file, a renamed file, or a library the game links against.
• catalog: A first resource viewer: every file of a game, what it is, and what the project's decoders make of it, as a static HTML page for people and a JSON manifest for agents.
• fields: Find the fields in many messages or records of one kind: which byte offsets are constant, which count up, which hold the length, which pick the kind of message, and which vary.
• markers: Mark the fields of a data file with unique values, then find where the game puts each one in memory.
• rawimage: Turn raw pixel data into a PNG: a dump of video memory, a screen saved by a DOS game, or a block of a data file that may hold a picture.
• reader: A binary reader (little-endian, with big-endian and variable-length reads) for prototyping decoders of a game's data files.
• shaders: Find the shaders inside a game's files: compiled Direct3D 9 and 10-12 shaders, SPIR-V, and shader source text in GLSL, HLSL and the OpenGL ARB assembly.
• shadetable: Read a shade, fog or remap table of a palette game: for each level, the colour that each palette colour turns into.
• skeleton: Find skeletons and poses in a model or animation file: arrays of parent indices, and runs of unit quaternions.
• timestamps: Decode a number as every common kind of timestamp, or find timestamps inside a file.
• validate: Run a decoder over every file of a type and report which files fail or leave bytes unread.
• xorscan: Find the key of a simply obfuscated data file: XOR with one byte or a repeating key, adding a byte, a key that steps by a fixed amount each byte, or each byte XORed with the one before.
• dosbox-run: Run a DOS program under DOSBox-X from the command line: the whole original game, for ground truth that a single emulated function cannot give (frames, demos, file access, timing).