locate

Tool tools/llm/locate.py (program, Models). Find a screen element, or a reference screen, in a capture with a vision model.

Run it with ./gameframe run locate ARGS or python3 tools/llm/locate.py. Source on GitHub.

Its own help

This is the help text in the tool itself, as ./gameframe tool locate prints it.

Find a screen element, or a reference screen, in a capture with a vision model.

Usage:
  locate.py NOW --find "the Exit button" [--units F]
  locate.py NOW --ref REF [--save FILE.json]

  NOW     a capture of what is shown now (PNG, JPEG, GIF or WebP)
  --find  the element to find, in words. Prints its centre in NOW's pixels, or "not found"
  --ref   a reference picture of the screen. Prints where it sits in NOW as an offset and a
          scale, like tools/live/align.py, and --save writes the same JSON, so
          `align.py map FILE.json X Y` gives click points from it
  --units divide the point given out by F (2 for a Retina capture's points)

The model gives positions as fractions of the picture's width and height, so they hold
whatever size the API scales the picture to. A position from a model is good to a few pixels.
For an exact offset, run tools/live/align.py first: it measures, and this tool guesses. Use
this one when align.py reports a weak match: another menu is open, the theme changed, or a
dialog covers part of the screen.

It uses the configured model API (client.py): one call with one or two pictures, about 2,000 to
4,000 input tokens. A picture must be under 5 MB. The selected model must accept images.

What it does not do: click. Give the point to winput or the VM's input tool yourself.

Named by

Skills: re-observe.