align

Tool tools/live/align.py (program, The running game). Line up a reference picture of a screen with a capture of the screen now: offset and scale.

Run it with ./gameframe run align ARGS or python3 tools/live/align.py. Source on GitHub.

Its own help

This is the help text in the tool itself, as ./gameframe tool align prints it.

Line up a reference picture of a screen with a capture of the screen now: offset and scale.

Usage:
  align.py find REF NOW [--point X,Y ...] [--scales MIN:MAX] [--stretch] [--save FILE.json]
                        [--check OUT.png] [--units F]
  align.py map FILE.json X Y [--inverse] [--units F]

  REF       the screen as you know it: a frame from the port, an earlier capture, a screenshot
            whose button positions you have (PNG)
  NOW       a capture of what is shown now: the VM's window with its frame, a stream, the
            whole desktop (PNG)
  --point   a point in REF (a button from the port's probe) to give in NOW's pixels; repeat it
  --scales  the range of scales to try, REF to NOW (default 0.25:4)
  --stretch let the vertical scale differ from the horizontal one (a stretched stream)
  --save    write the result as JSON, for `map` and for the next click
  --check   write NOW with REF's outline and each point drawn on it, to look at before clicking
  --units   divide the points given out by F: 2 turns a Retina capture's pixels into the
            points that macOS input tools use
  map       give a point of REF in NOW (or, with --inverse, a point of NOW in REF) from a saved
            result

The result is `now = offset + ref * scale`, for x and y. It is found in three steps: every
scale and position on a small copy of both pictures, then the best three candidates on a
larger copy, then the best one on samples of the full pictures. The score is the normalised
correlation of the two pictures where they overlap, from -1 to 1. Below 0.5 the match is weak,
and the tool says so.

Why: a click point taken from the port, or from an earlier capture, misses when the window
has a frame, a title bar, a border or a different size. The first click lands on another
button. Align once, then map every point.

What it does not do: rotation, perspective, or a screen whose content has changed a lot since
REF (another menu, a moved window inside the game). When the pictures differ too much, ask a
model to find the element: tools/llm/locate.py. Standard library only; PNG through
tools/formats/pngio.py (8-bit, not interlaced).

Named by

Skills: re-observe.